Picture labeling method and device

Abstract

The application discloses a picture labeling method and device, and belongs to the field of data labeling. The method comprises the following steps: obtaining a labeling result of a second picture before a first picture to be labeled in a picture sequence; determining the position of a labeling target in the N pictures before the first picture in the labeling result, determining the position change rule of the labeling target according to the position of the labeling target in the N pictures; and determining the predicted position of the labeling target in the first picture according to the position change rule of the labeling target, and then the labeling target in the first picture can be labeled according to the predicted position. In this way, the labeling efficiency and accuracy are improved, the algorithm is simpler, is more suitable for real-time prediction scenarios, and has lower requirements on hardware resources.

Description

Technical Field

[0001] This application relates to the field of data annotation, and in particular to a method and apparatus for image annotation. Background Technology

[0002] In the field of machine learning, to obtain training data related to image recognition, it is necessary to first label a sequence of images, and then use each image in the sequence and its corresponding labeling results as training data. For example, the labeling results can include the location, type, or color of the labeled target. The labeled target is a pre-specified object that needs to be labeled, such as a person or a vehicle.

[0003] In related technologies, in order to improve the efficiency of image annotation, when annotating an image sequence, for each image to be annotated in the image sequence, an object detection model can be used to detect objects in each image to identify the location of the object to be annotated in each image. Then, based on the identified location, a bounding box of the object to be annotated is drawn at the corresponding location in the image, and annotation information such as object identifier, type or color is added to the object to be annotated in the image.

[0004] However, due to the complexity of the object detection model's algorithm, the labeling of the recognition results using the object detection model is time-consuming and has high requirements for the hardware environment, thus having certain limitations. Summary of the Invention

[0005] This application provides an image annotation method and apparatus, which can solve the problems of related technologies, such as the long time consumption and high hardware requirements for annotation based on the recognition results of object detection models. The technical solution is as follows:

[0006] On the one hand, an image annotation method is provided, the method comprising:

[0007] For the first image to be labeled in the image sequence, obtain the labeling result of the second image located before the first image, the labeling result including the location of the labeled target in the second image;

[0008] Determine the position of the labeled target in N images preceding the first image, and determine the position change pattern of the labeled target based on the position of the labeled target in the N images, where N is an integer greater than 1;

[0009] Based on the positional change pattern of the labeled target, the predicted position of the labeled target in the first image is determined;

[0010] The first icon is labeled based on the predicted location.

[0011] Optionally, the labeled targets include M targets, where M is a positive integer;

[0012] The step of determining the position change pattern of the labeled target based on its position in the N images includes:

[0013] For the first target among the M targets, the position coordinates of the first target in the N images are linearly fitted to obtain a linear regression equation with the image order as the independent variable and the position coordinates of the first target as the dependent variable. The first target is any one of the M targets.

[0014] Determining the predicted position of the labeled target in the first image based on the position change pattern of the labeled target includes:

[0015] Based on the order of the first image in the image sequence, the predicted position coordinates of the first target in the first image are determined by the linear regression equation.

[0016] Optionally, determining the position of the labeled target in the N images preceding the first image includes:

[0017] Determine the position of the target bounding box of the first target in the N images, and obtain the position coordinates of N target bounding boxes that correspond one-to-one with the N images;

[0018] The step of linearly fitting the position coordinates of the first target in the N images to obtain a linear regression equation with image order as the independent variable and the position coordinates of the first target as the dependent variable includes:

[0019] The linear regression equation is obtained by linearly fitting the position coordinates of the N target boxes that correspond one-to-one with the N images.

[0020] Optionally, the position coordinates of each of the N target boxes include the coordinates of the first vertex and the second vertex of each target box;

[0021] The step of linearly fitting the position coordinates of the N target boxes corresponding one-to-one with the N images to obtain the linear regression equation includes:

[0022] Linear fitting is performed on the X-axis coordinates of the first vertex coordinates of the N target boxes that correspond one-to-one with the N images to obtain a first linear regression equation with image sorting as the independent variable and the X-axis coordinates of the first vertex of the target box of the first target as the dependent variable.

[0023] Linear fitting is performed on the Y-axis coordinates of the first vertex coordinates of the N target boxes that correspond one-to-one with the N images to obtain a second linear regression equation with the image order as the independent variable and the Y-axis coordinates of the first vertex of the target box of the first target as the dependent variable.

[0024] Linear fitting is performed on the X-axis coordinates of the second vertices of the N target boxes that correspond one-to-one with the N images to obtain a third linear regression equation with image sorting as the independent variable and the X-axis coordinates of the second vertices of the target box of the first target as the dependent variable.

[0025] Linear fitting is performed on the Y-axis coordinates of the second vertices of the N target boxes that correspond one-to-one with the N images to obtain a fourth linear regression equation with image sorting as the independent variable and the Y-axis coordinates of the second vertices of the target box of the first target as the dependent variable.

[0026] Optionally, determining the predicted position coordinates of the first target in the first image based on the order of the first image in the image sequence using the linear regression equation includes:

[0027] Based on the order of the first image in the image sequence, the predicted X-axis coordinates and predicted Y-axis coordinates of the first vertex of the target box of the first target in the first image, and the predicted X-axis coordinates and predicted Y-axis coordinates of the second vertex of the target box of the first target are determined by the first linear regression equation, the second linear regression equation, the third linear regression equation, and the fourth linear regression equation, respectively.

[0028] Optionally, labeling the first icon based on the predicted location includes:

[0029] Based on the predicted location, the target objects existing in the predicted location are labeled;

[0030] or,

[0031] The predicted location is highlighted in the first image to remind the annotator to mark the target objects present at the predicted location.

[0032] Optionally, the predicted position of the labeled target in the first image includes the predicted position coordinates of the target bounding box of the labeled target;

[0033] Highlighting the predicted location in the first image includes:

[0034] Based on the predicted location coordinates, draw the target bounding box of the labeled target in the first image.

[0035] Optionally, after drawing the target bounding box of the labeled target in the first image, the method further includes:

[0036] Based on the target identifiers of the labeled targets included in the annotation results of the second image, target identifiers are added to the labeled targets existing in the first image.

[0037] Optionally, before determining the position of the labeled target in the N images preceding the first image, the method further includes:

[0038] If the labeled target is detected to appear consecutively in N-1 images, then the step of determining the position of the labeled target in the N images preceding the first image is performed, where the N images are all images other than the second image.

[0039] If the target to be labeled is detected not to appear consecutively in N-1 images, then no auxiliary labeling will be performed on the first image.

[0040] On the other hand, an image annotation device is provided, the device comprising:

[0041] The acquisition module is used to acquire the annotation result of a second image located before the first image in an image sequence, wherein the annotation result includes the position of the annotation target in the second image;

[0042] The first determining module is used to determine the position of the labeled target in N images preceding the first image, and to determine the position change pattern of the labeled target based on the position of the labeled target in the N images, where N is an integer greater than 1;

[0043] The second determining module is used to determine the predicted position of the labeled target in the first image based on the position change pattern of the labeled target;

[0044] The annotation module is used to annotate the first icon based on the predicted location.

[0045] Optionally, the labeled targets include M targets, where M is a positive integer;

[0046] The first determining module is used for:

[0047] For the first target among the M targets, the position coordinates of the first target in the N images are linearly fitted to obtain a linear regression equation with the image order as the independent variable and the position coordinates of the first target as the dependent variable. The first target is any one of the M targets.

[0048] The second determining module is used for:

[0049] Based on the order of the first image in the image sequence, the predicted position coordinates of the first target in the first image are determined by the linear regression equation.

[0050] Optionally, the first determining module is used to:

[0051] Determine the position of the target bounding box of the first target in the N images, and obtain the position coordinates of N target bounding boxes that correspond one-to-one with the N images;

[0052] The linear regression equation is obtained by linearly fitting the position coordinates of the N target boxes that correspond one-to-one with the N images.

[0053] Optionally, the position coordinates of each of the N target boxes include the coordinates of the first vertex and the second vertex of each target box;

[0054] The first determining module is used for:

[0055] Linear fitting is performed on the X-axis coordinates of the first vertex coordinates of the N target boxes that correspond one-to-one with the N images to obtain a first linear regression equation with image sorting as the independent variable and the X-axis coordinates of the first vertex of the target box of the first target as the dependent variable.

[0056] Linear fitting is performed on the Y-axis coordinates of the first vertex coordinates of the N target boxes that correspond one-to-one with the N images to obtain a second linear regression equation with the image order as the independent variable and the Y-axis coordinates of the first vertex of the target box of the first target as the dependent variable.

[0057] Linear fitting is performed on the X-axis coordinates of the second vertices of the N target boxes that correspond one-to-one with the N images to obtain a third linear regression equation with image sorting as the independent variable and the X-axis coordinates of the second vertices of the target box of the first target as the dependent variable.

[0058] Linear fitting is performed on the Y-axis coordinates of the second vertices of the N target boxes that correspond one-to-one with the N images to obtain a fourth linear regression equation with image sorting as the independent variable and the Y-axis coordinates of the second vertices of the target box of the first target as the dependent variable.

[0059] Optionally, the second determining module is used to:

[0060] Based on the order of the first image in the image sequence, the predicted X-axis coordinates and predicted Y-axis coordinates of the first vertex of the target box of the first target in the first image, and the predicted X-axis coordinates and predicted Y-axis coordinates of the second vertex of the target box of the first target are determined by the first linear regression equation, the second linear regression equation, the third linear regression equation, and the fourth linear regression equation, respectively.

[0061] Optionally, the annotation module includes:

[0062] A labeling unit is used to label the target objects existing in the predicted location based on the predicted location;

[0063] or,

[0064] The display unit is used to highlight the predicted location in the first image to remind the annotator to annotate the target present in the predicted location.

[0065] Optionally, the predicted position of the labeled target in the first image includes the predicted position coordinates of the target bounding box of the labeled target;

[0066] The display unit is used for:

[0067] Based on the predicted location coordinates, draw the target bounding box of the labeled target in the first image.

[0068] Optionally, the annotation module is further configured to:

[0069] Based on the target identifiers of the labeled targets included in the annotation results of the second image, target identifiers are added to the labeled targets existing in the first image.

[0070] Optionally, the device further includes:

[0071] The first triggering module is used to trigger the first determining module to determine the position of the labeled target in the N images preceding the first image if the labeled target is detected to appear consecutively in N-1 images. The N images are the other images in the N images excluding the second image.

[0072] The second triggering module is used to prevent auxiliary annotation of the first image if it is detected that the labeled target does not appear consecutively in N-1 images.

[0073] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory for storing computer programs, and the processor for executing the programs stored in the memory to implement the steps of any of the above-described image annotation methods.

[0074] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of any of the above-described image annotation methods.

[0075] On the other hand, a computer program product containing instructions is provided that, when run on a computer, causes the computer to perform the steps of the image annotation method described above.

[0076] The technical solution provided in this application can bring at least the following beneficial effects:

[0077] When labeling image sequences, for each image to be labeled, the labeling result of the previous image can be obtained first. Then, based on the position of the target image in the previous N images, the positional change pattern of the target image can be determined, thereby predicting the position of the target image in the first image. The predicted position is then used to assist in labeling the target image in the first image, thus improving labeling efficiency and accuracy. Moreover, this labeling algorithm is simpler and faster than object detection algorithms, making it more suitable for real-time prediction scenarios. It also has lower hardware resource requirements and lower deployment costs. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0079] Figure 1 This is a flowchart of an image annotation method provided in an embodiment of this application;

[0080] Figure 2 This is a flowchart of another image annotation method provided in the embodiments of this application;

[0081] Figure 3 This is a schematic diagram of the structure of an image annotation device provided in an embodiment of this application;

[0082] Figure 4 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0084] Before providing a detailed description of this application, let's first introduce its application scenarios.

[0085] The image annotation method provided in this application is applied to scenarios involving the annotation of image sequences. For example, it can assist annotators during manual annotation, reducing repetitive work, accelerating annotation speed, lowering the risk of missed annotations, and improving annotation accuracy.

[0086] For example, before annotators label an image, the image annotation method provided in this application can be used to predict the location of the target to be labeled in the current image, and the predicted location can be highlighted to remind annotators to check whether the highlighted target is a real and valid target for manual verification. After the manual verification is passed, other annotation information can be added to the highlighted target. For example, this method can be used to predict the location of the bounding box of the target to be labeled in the current image, and draw the bounding box of the target in the current image based on the predicted location.

[0087] It should be noted that the image sequence to be labeled can be individual images arranged in sequence, or it can be video frames from a video. That is, the method provided in this application can provide auxiliary labeling for multiple individual images, or it can provide auxiliary labeling for video frames from a video.

[0088] It should also be noted that, because the image annotation method provided in this application embodiment has a simple algorithm, it has a short processing time, making it more suitable for real-time prediction scenarios. Furthermore, it requires lower hardware resources and has a lower deployment environment cost. For example, the image annotation method provided in this application embodiment only requires a regular CPU and does not require a GPU.

[0089] The image annotation method provided in the embodiments of this application will be explained in detail below.

[0090] Figure 1 This is a flowchart illustrating an image annotation method provided in an embodiment of this application. This method is applied in computer devices, such as the CPU of a computer device. Please refer to... Figure 1 The method includes the following steps:

[0091] Step 101: For the first image to be labeled in the image sequence, obtain the labeling result of the second image located before the first image. The labeling result includes the location of the labeled targets in the second image.

[0092] The second image is the image preceding the first image. When it is necessary to annotate the first image, the annotation results of the preceding image can be obtained first. Based on the annotation results of the preceding image, the location of the target to be annotated in the first image can be predicted, thus assisting in the annotation of the first target.

[0093] The first image contains pre-defined targets that need to be labeled, such as people or vehicles. The second image's labeling results include the locations of the labeled targets within that image, which can be represented by coordinates. For example, the second image's labeling results may include the coordinates of the bounding boxes of the labeled targets. Additionally, the second image's labeling results may also include the target identifier, type, color, or size of the labeled targets. The target identifier can be the target name or target ID, etc.

[0094] Step 102: Determine the position of the labeled target in the N images preceding the first image. Based on the position of the labeled target in these N images, determine the position change pattern of the labeled target, where N is an integer greater than 1.

[0095] Here, the N images are the first N images of the first image, and N can be preset, for example, N can be 10 or 15, etc. The position change pattern of the labeled target can be a position change curve or a linear fitting equation that can indicate the position change of the labeled target.

[0096] As an example, the labeled targets can include M targets, where M is a positive integer. That is, the labeled targets can include one or more targets, and the target types of the multiple targets can be the same or different. When determining the position change pattern of the labeled targets, for each of the M targets included in the labeled targets, the position of each target in the N images preceding the first image can be determined. Based on the position of each target in these N images, the position change pattern of each target is determined, and based on the position change pattern of each target, the predicted position of each target in the first image can be determined.

[0097] As an example, for the first target among M targets, we can perform a linear regression on the position coordinates of the first target in N images to obtain a linear regression equation with image order as the independent variable and the position coordinates of the first target as the dependent variable. Here, the first target can be any one of the M targets.

[0098] For example, if the location of the labeled target is the location of the target bounding box, then for the first target among M targets, we can first determine the location of the target bounding box of the first target in N images, obtain the position coordinates of the N target bounding boxes corresponding one-to-one with the N images, and then perform linear fitting on the position coordinates of the N target bounding boxes corresponding one-to-one with the N images to obtain a linear regression equation.

[0099] As an example, the position coordinates of each of the N target boxes include the coordinates of the first vertex and the second vertex of each target box, where the first vertex and the second vertex are different vertices. For example, the first vertex is the top left corner of the target box and the second vertex is the bottom right corner of the target box; or, the first vertex is the top right corner of the target box and the second vertex is the bottom left corner of the target box.

[0100] When performing linear fitting on the position coordinates of the target boxes, we can perform linear fitting on the first vertex coordinates of the N target boxes that correspond one-to-one with the N images to obtain a linear regression equation with the image order as the independent variable and the position coordinates of the first vertex of the target box of the first target as the dependent variable. We can also perform linear fitting on the second vertex coordinates of the N target boxes that correspond one-to-one with the N images to obtain a linear regression equation with the image order as the independent variable and the position coordinates of the second vertex of the target box of the first target as the dependent variable.

[0101] As an example, the coordinates of the first vertex and the second vertex can be coordinates in a Cartesian coordinate system constructed based on the first image, which includes mutually perpendicular X-axis and Y-axis. For example, the Cartesian coordinate system could be a Cartesian coordinate system with the bottom left corner of the first image as the origin and two mutually perpendicular sides as the X-axis and Y-axis. For example, the coordinates of the first vertex could include X-axis coordinates and Y-axis coordinates, and the coordinates of the second vertex could also include X-axis coordinates and Y-axis coordinates.

[0102] Furthermore, the X-axis and Y-axis coordinates of the first vertex can be fitted separately, as can the X-axis and Y-axis coordinates of the second vertex. For example, the process of linearly fitting the position coordinates of the N target boxes corresponding one-to-one with N images to obtain a linear regression equation can include the following steps:

[0103] 1) Perform linear fitting on the X-axis coordinates of the first vertex coordinates of the N target boxes that correspond one-to-one with the N images to obtain the first linear regression equation with the image order as the independent variable and the X-axis coordinates of the first vertex of the target box of the first target as the dependent variable.

[0104] 2) Perform linear fitting on the Y-axis coordinates of the first vertex coordinates of the N target boxes that correspond one-to-one with the N images to obtain a second linear regression equation with the image order as the independent variable and the Y-axis coordinate of the first vertex of the target box of the first target as the dependent variable.

[0105] 3) Perform linear fitting on the X-axis coordinates of the second vertices of the N target boxes that correspond one-to-one with the N images to obtain a third linear regression equation with the image order as the independent variable and the X-axis coordinates of the second vertices of the target box of the first target as the dependent variable.

[0106] 4) Perform linear fitting on the Y-axis coordinates of the second vertices of the N target boxes that correspond one-to-one with the N images to obtain the fourth linear regression equation with the image order as the independent variable and the Y-axis coordinates of the second vertices of the target box of the first target as the dependent variable.

[0107] It should be noted that the first, second, third, and fourth linear regression equations mentioned above are all linear equations in one variable.

[0108] Step 103: Based on the positional change pattern of the labeled target, determine the predicted position of the labeled target in the first image.

[0109] As an example, if the position change pattern of the labeled target is a linear regression equation that can indicate the position change of the labeled target, then the predicted position coordinates of the first target in the first image can be determined by the linear regression equation based on the order of the first image in the image sequence.

[0110] For example, the order of the first image in the image sequence is substituted into the linear regression equation, and the position coordinates of the first target are calculated through the linear regression equation. The position coordinates of the first target are then used as the predicted position coordinates of the first target in the first image.

[0111] As an example, if the linear regression equation includes the first, second, third, and fourth linear regression equations mentioned above, then based on the order of the first image in the image sequence, the predicted X-axis and Y-axis coordinates of the first vertex of the target box of the first target in the first image, as well as the predicted X-axis and Y-axis coordinates of the second vertex of the target box of the first target, can be determined by the first, second, third, and fourth linear regression equations, respectively.

[0112] That is, based on the order of the first image in the image sequence, the predicted X-axis coordinate of the first vertex of the target box of the first target in the first image can be determined by the first linear regression equation; the predicted Y-axis coordinate of the first vertex can be determined by the second linear regression equation; the predicted X-axis coordinate of the second vertex can be determined by the third linear regression equation; and the predicted Y-axis coordinate of the second vertex can be determined by the fourth linear regression equation.

[0113] It should be noted that the X-axis and Y-axis coordinates of the first vertex, as well as the X-axis and Y-axis coordinates of the second vertex, can indicate the predicted position coordinates of the bounding box of the first target. Alternatively, the predicted position coordinates of the bounding box of the first target can also be determined based on the X-axis and Y-axis coordinates of the first vertex and the second vertex.

[0114] Step 104: Based on the predicted location, label the first icon.

[0115] In this embodiment of the application, the first icon can be automatically labeled based on the predicted location, or the first icon can be labeled by an assisted labeler based on the predicted location.

[0116] As an example, labeling the first icon based on this predicted location can be implemented in the following two ways:

[0117] The first possible implementation is to label the targets that exist at the predicted location.

[0118] In other words, the target in the predicted location can be directly identified as the target to be labeled, and the target can be labeled. This method has high labeling efficiency, reduces manual intervention, and lowers labeling costs.

[0119] As an example, the operation of labeling targets includes: adding target identifiers to the labeled targets existing in the first image based on the target identifiers included in the labeling results of the second image. That is, after determining the predicted location, the same target identifier as the same labeled target in the previous image can be directly added to the labeled targets at the predicted location. This reduces repetitive work for labelers and speeds up the labeling process.

[0120] The second possible implementation is to highlight the predicted location in the first image to remind the annotators to annotate the targets present at that predicted location.

[0121] In other words, by highlighting the predicted locations in the first image, annotators can better see the targets within those locations and be reminded to annotate them. This provides an auxiliary annotation method that helps annotators label images, reducing the risk of missing targets and improving annotation efficiency and accuracy.

[0122] Highlighting the predicted location refers to displaying the area corresponding to that predicted location with special effects, so that annotators can intuitively see the annotation targets present at that location. For example, highlighting the predicted location can be done by displaying it with a specific color, highlighting it, or outlining it.

[0123] As an example, the bounding box of the target can be drawn in the first image based on the predicted location coordinates. In this way, the annotator can see the bounding box of the target before annotating, which helps the annotator to annotate the target without having to search for the target in the first image by eye, reducing the visual workload of the annotator and speeding up the annotation process.

[0124] Highlighting predicted locations reminds annotators to label targets within those locations, reducing the risk of missing any. Furthermore, highlighting predicted locations prompts annotators to verify that the highlighted targets are valid for manual verification. If a target is valid, additional annotation information is added; otherwise, the target's bounding box is removed, and it is not labeled. This reduces labeling errors and improves accuracy.

[0125] As an example, after drawing the bounding box of the labeled target in the first image, target identifiers can be added to the labeled targets existing in the first image based on the target identifiers included in the annotation results of the second image. That is, after drawing the bounding box of the labeled target in the first image, the same target identifier as the same labeled target in the previous image can be directly added to the labeled target in the first image. This reduces repetitive work for annotators and speeds up the annotation process.

[0126] It should be noted that in the field of data annotation, the target identifier of the same target in different images in an image sequence must be the same. Whether they are the same target can be determined by whether the target identifier is consistent.

[0127] In this embodiment, when labeling an image sequence, for each image to be labeled, the labeling result of the previous image can be obtained first. Then, based on the position of the target image in the previous N images, the position change pattern of the target image is determined, thereby predicting the position of the target image in the first image. The predicted position is then used to assist in labeling the target image in the first image, thus improving labeling efficiency and accuracy. Furthermore, this labeling algorithm is simpler and faster than object detection algorithms, making it more suitable for real-time prediction scenarios. It also has lower hardware resource requirements and lower deployment costs.

[0128] Figure 2 This is a flowchart of another image annotation method provided in the embodiments of this application, such as... Figure 2 As shown, the method includes the following steps:

[0129] Step 201: For the first image to be labeled in the image sequence, check whether the first image already has a corresponding labeling result.

[0130] As an example, after importing the image sequence to be labeled, each image in the image sequence can be sequentially labeled according to the method provided in the embodiments of this application.

[0131] Step 202: If the first image already has corresponding annotation results, then load the annotation results of the first image.

[0132] In other words, if the first image has already been labeled and the labeling results are available, then there is no need to label the first image again; the existing labeling results can be loaded and used directly.

[0133] The first image already has corresponding annotation results, which can be obtained manually or by equipment. This application embodiment does not limit this.

[0134] As an example, for a sequence of images to be labeled, the labeler can manually label the first N images first, so that the device can use the labeling results of the first N images to assist in labeling the subsequent images, thus improving the labeling accuracy.

[0135] Step 203: If the first image does not have a corresponding annotation result, then obtain the annotation result of the second image located before the first image.

[0136] In other words, if the first image has not yet been labeled, the labeling result of the previous image can be obtained so that the current first icon can be labeled based on the labeling result of the previous image.

[0137] If the first image has a corresponding annotation result, it means that the first icon has been annotated. In this case, the method provided in the embodiments of this application can be used to annotate the first icon, and the already annotated annotation result can be used directly. Alternatively, the method provided in the embodiments of this application can be used to annotate the first icon, and the existing annotation result can be updated based on the subsequent annotation results.

[0138] Step 204: Check if the annotation result of the second image is empty.

[0139] If the annotation result of the second image is not empty, then proceed to step 205 below; if the annotation result of the second image is empty, then no auxiliary annotation is performed on the first image.

[0140] Among them, auxiliary annotation refers to highlighting the predicted position of the target in the first image after determining the predicted position of the target in the first icon, so as to remind the annotator to annotate the target that exists in the predicted position.

[0141] In other words, if the previous image does not have any annotation results, then the current first image does not need to be annotated, but rather the annotation is done manually by the annotator.

[0142] Step 205: If the annotation result of the second image is not empty, then determine the annotation targets in the annotation result of the second image.

[0143] In other words, if the previous image has annotations, the annotated targets can be determined from the annotations of the second image. These annotated targets can include one or more targets.

[0144] Step 206: Detect whether the labeled target appears consecutively in N-1 images, where N-1 images are the other images besides the second image among the first N images of the first image.

[0145] In other words, it can detect whether the annotation result already marked in the previous icon appears consecutively in the previous N icons, that is, whether the annotation result appears in every one of the previous N images.

[0146] If the target appears consecutively in N-1 images, proceed to step 208 below; if the target does not appear consecutively in N-1 images, no auxiliary annotation is performed on the first image.

[0147] Step 207: If the labeled target appears consecutively in N-1 images, determine the position change pattern of the labeled target based on its position in the first N images, and determine the predicted position of the labeled target in the first image based on the position change pattern of the labeled target.

[0148] Step 208: Highlight the predicted location in the first image to remind the annotators to annotate the targets present at the predicted location.

[0149] It should be noted that the specific implementation of steps 207-208 can refer to the relevant description of steps 102-104 above, and this application embodiment does not limit it in this way.

[0150] It should also be noted that, Figure 2 This embodiment is merely an optional embodiment provided by this application, and some steps therein may be optional. Those skilled in the art can adapt it accordingly. Figure 2 The embodiments may be modified accordingly, as long as they can achieve the purpose of auxiliary annotation.

[0151] In this embodiment, for the first image to be labeled in an image sequence, the labeling results of the second image preceding the first image are obtained. Then, based on the position of the labeled target in the N images preceding the first image, the positional change pattern of the labeled target is determined, thereby predicting the position of the labeled target in the first image. The predicted position is then highlighted in the first image, reminding the labelers to label the highlighted target. This provides an auxiliary labeling method that assists labelers in labeling images, reducing the risk of missed labeling and improving labeling efficiency and accuracy. Moreover, this auxiliary labeling algorithm is simpler than object detection algorithms, resulting in shorter processing time, making it more suitable for real-time prediction scenarios. It also has lower hardware resource requirements and lower deployment costs.

[0152] Figure 3 This is a schematic diagram of an image annotation device provided in an embodiment of this application. The device can be implemented as part or all of a computer device by software, hardware, or a combination of both. Please refer to... Figure 3 The device includes:

[0153] The acquisition module 301 is used to acquire the annotation result of a second image located before the first image in an image sequence for a first image to be annotated, wherein the annotation result includes the position of the annotation target in the second image;

[0154] The first determining module 302 is used to determine the position of the labeled target in N images located before the first image, and to determine the position change pattern of the labeled target based on the position of the labeled target in the N images, where N is an integer greater than 1;

[0155] The second determining module 303 is used to determine the predicted position of the labeled target in the first image based on the position change pattern of the labeled target;

[0156] The annotation module 304 is used to annotate the first icon based on the predicted position.

[0157] Optionally, the labeled targets include M targets, where M is a positive integer;

[0158] The first determining module 302 is used for:

[0159] For the first target among the M targets, the position coordinates of the first target in the N images are linearly fitted to obtain a linear regression equation with the image order as the independent variable and the position coordinates of the first target as the dependent variable. The first target is any one of the M targets.

[0160] The second determining module 303 is used for:

[0161] Based on the order of the first image in the image sequence, the predicted position coordinates of the first target in the first image are determined by the linear regression equation.

[0162] Optionally, the first determining module 302 is used to:

[0163] Determine the position of the target bounding box of the first target in the N images, and obtain the position coordinates of N target bounding boxes that correspond one-to-one with the N images;

[0164] The linear regression equation is obtained by linearly fitting the position coordinates of the N target boxes that correspond one-to-one with the N images.

[0165] Optionally, the position coordinates of each of the N target boxes include the coordinates of the first vertex and the second vertex of each target box;

[0166] The first determining module 302 is used for:

[0167] Linear fitting is performed on the X-axis coordinates of the first vertex coordinates of the N target boxes that correspond one-to-one with the N images to obtain a first linear regression equation with image sorting as the independent variable and the X-axis coordinates of the first vertex of the target box of the first target as the dependent variable.

[0168] Linear fitting is performed on the Y-axis coordinates of the first vertex coordinates of the N target boxes that correspond one-to-one with the N images to obtain a second linear regression equation with the image order as the independent variable and the Y-axis coordinates of the first vertex of the target box of the first target as the dependent variable.

[0169] Linear fitting is performed on the X-axis coordinates of the second vertices of the N target boxes that correspond one-to-one with the N images to obtain a third linear regression equation with image sorting as the independent variable and the X-axis coordinates of the second vertices of the target box of the first target as the dependent variable.

[0170] Linear fitting is performed on the Y-axis coordinates of the second vertices of the N target boxes that correspond one-to-one with the N images to obtain a fourth linear regression equation with image sorting as the independent variable and the Y-axis coordinates of the second vertices of the target box of the first target as the dependent variable.

[0171] Optionally, the second determining module 303 is used to:

[0172] Based on the order of the first image in the image sequence, the predicted X-axis coordinates and predicted Y-axis coordinates of the first vertex of the target box of the first target in the first image, and the predicted X-axis coordinates and predicted Y-axis coordinates of the second vertex of the target box of the first target are determined by the first linear regression equation, the second linear regression equation, the third linear regression equation, and the fourth linear regression equation, respectively.

[0173] Optionally, the annotation module 304 includes:

[0174] A labeling unit is used to label the target objects existing in the predicted location based on the predicted location;

[0175] or,

[0176] The display unit is used to highlight the predicted location in the first image to remind the annotator to annotate the target present in the predicted location.

[0177] Optionally, the predicted position of the labeled target in the first image includes the predicted position coordinates of the target bounding box of the labeled target;

[0178] The display unit is used for:

[0179] Based on the predicted location coordinates, draw the target bounding box of the labeled target in the first image.

[0180] Optionally, the annotation module is further configured to:

[0181] Based on the target identifiers of the labeled targets included in the annotation results of the second image, target identifiers are added to the labeled targets existing in the first image.

[0182] Optionally, the device further includes:

[0183] The first triggering module is used to trigger the first determining module to determine the position of the labeled target in the N images preceding the first image if the labeled target is detected to appear consecutively in N-1 images. The N images are the other images in the N images excluding the second image.

[0184] The second triggering module is used to prevent auxiliary annotation of the first image if it is detected that the labeled target does not appear consecutively in N-1 images.

[0185] In this embodiment, when labeling an image sequence, for each image to be labeled, the labeling result of the previous image can be obtained first. Then, based on the position of the target image in the previous N images, the position change pattern of the target image is determined, thereby predicting the position of the target image in the first image. The predicted position is then used to assist in labeling the target image in the first image, thus improving labeling efficiency and accuracy. Furthermore, this labeling algorithm is simpler and faster than object detection algorithms, making it more suitable for real-time prediction scenarios. It also has lower hardware resource requirements and lower deployment costs.

[0186] It should be noted that the image annotation device provided in the above embodiments is only illustrated by the division of the above functional modules when annotating images. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image annotation device and the image annotation method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0187] Figure 4 This is a structural block diagram of a computer device 400 provided in an embodiment of this application. The computer device 400 can be an electronic device such as a mobile phone, tablet computer, smart TV, multimedia playback device, wearable device, desktop computer, or server. The computer device 400 can be used to implement the image annotation method provided in the above embodiments.

[0188] Typically, computer device 400 includes a processor 401 and a memory 402.

[0189] Processor 401 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0190] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 402 are used to store at least one instruction, which is executed by the processor 401 to implement the image annotation method provided in the method embodiments of this application.

[0191] In some embodiments, the computer device 400 may also optionally include a peripheral device interface 403 and at least one peripheral device. The processor 401, memory 402, and peripheral device interface 403 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 403 via a bus, signal line, or circuit board. Specifically, the peripheral device may include at least one of a display screen 404, audio circuitry 405, communication interface 406, and power supply 407.

[0192] Those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on computer device 400, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0193] In an exemplary embodiment, a computer-readable storage medium is also provided, on which instructions are stored, which, when executed by a processor, implement the above-described image annotation method.

[0194] In an exemplary embodiment, a computer program product is also provided, which, when executed, is used to implement the above-described image annotation method.

[0195] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A picture labeling method characterized by, The method is applied to a central processing unit (CPU), and the method comprises: For a first picture to be labeled in a picture sequence, obtaining a labeling result of a second picture located before the first picture, the labeling result comprising positions of labeling targets existing in the second picture, the labeling targets comprising M targets, M being a positive integer; For a first target in the M targets, if it is detected that the first target continuously appears in N-1 pictures, determining positions of target boxes in N pictures before the first target, to obtain position coordinates of N target boxes corresponding to the N pictures one by one; wherein the position coordinates of each target box in the N target boxes comprise first vertex coordinates and second vertex coordinates of each target box, the first vertex and the second vertex being different vertices; the first vertex and the second vertex are the upper left corner and the lower right corner of the target box respectively, or the first vertex and the second vertex are the upper right corner and the lower left corner of the target box respectively; the first target is any one of the M targets, the N-1 pictures are other pictures in the N pictures except the second picture, and N is an integer greater than 1; Linearly fitting X-axis coordinates of the first vertex coordinates of the N target boxes corresponding to the N pictures one by one, to obtain a first linear regression equation with picture order as the independent variable and the X-axis coordinates of the first vertex of the target box of the first target as the dependent variable; Linearly fitting Y-axis coordinates of the first vertex coordinates of the N target boxes corresponding to the N pictures one by one, to obtain a second linear regression equation with picture order as the independent variable and the Y-axis coordinates of the first vertex of the target box of the first target as the dependent variable; Linearly fitting X-axis coordinates of the second vertex coordinates of the N target boxes corresponding to the N pictures one by one, to obtain a third linear regression equation with picture order as the independent variable and the X-axis coordinates of the second vertex of the target box of the first target as the dependent variable; Linearly fitting Y-axis coordinates of the second vertex coordinates of the N target boxes corresponding to the N pictures one by one, to obtain a fourth linear regression equation with picture order as the independent variable and the Y-axis coordinates of the second vertex of the target box of the first target as the dependent variable; the first linear regression equation, the second linear regression equation, the third linear regression equation and the fourth linear regression equation are all monomial first-order equations; According to the order of the first picture in the picture sequence, the predicted X-axis coordinates and the predicted Y-axis coordinates of the first vertex of the target box of the first target in the first picture, and the predicted X-axis coordinates and the Y-axis coordinates of the second vertex of the target box of the first target are determined by the first linear regression equation, the second linear regression equation, the third linear regression equation and the fourth linear regression equation respectively. highlighting or marking out a contour. If it is detected that the first target does not continuously appear in the N-1 pictures, the first picture is not subjected to the auxiliary labeling.

2. The method of claim 1, wherein, After the step of highlighting the predicted position of the labeling target in the first picture, the method further comprises: adding a target identifier to the labeling target existing in the first picture according to a target identifier of the labeling target included in the labeling result of the second picture.

3. An image labeling apparatus characterized by comprising: The device is included in a central processing unit (CPU), and the device comprises: a first triggering module, configured to, for a first target in the M targets, if it is detected that the first target continuously appears in N-1 pictures, trigger a first determining module to perform a step of determining positions of target frames in N pictures before the first picture, the first target being any one of the M targets, the N-1 pictures being other pictures than the second picture in the N pictures, and the N being an integer greater than 1; and a first determining module, configured to, for the first target in the M targets, if it is detected that the first target continuously appears in N-1 pictures, determine positions of target frames in N pictures before the first picture. The first determining module is configured to, for a first target in the M targets, determine positions of target frames in N pictures before the first target in the first picture under triggering of the first triggering module, to obtain position coordinates of N target frames corresponding to the N pictures; wherein the position coordinates of each target frame in the N target frames include first vertex coordinates and second vertex coordinates of each target frame, the first vertex and the second vertex are different vertices; the first vertex and the second vertex are a top-left corner and a bottom-right corner of the target frame respectively, or the first vertex and the second vertex are a top-right corner and a bottom-left corner of the target frame respectively; X-axis coordinates of the first vertex coordinates of the N target frames corresponding to the N pictures are linearly fitted to obtain a first linear regression equation with picture order as an independent variable and X-axis coordinates of the first vertex of the target frame of the first target as a dependent variable; Y-axis coordinates of the first vertex coordinates of the N target frames corresponding to the N pictures are linearly fitted to obtain a second linear regression equation with picture order as an independent variable and Y-axis coordinates of the first vertex of the target frame of the first target as a dependent variable; X-axis coordinates of the second vertex coordinates of the N target frames corresponding to the N pictures are linearly fitted to obtain a third linear regression equation with picture order as an independent variable and X-axis coordinates of the second vertex of the target frame of the first target as a dependent variable; Y-axis coordinates of the second vertex coordinates of the N target frames corresponding to the N pictures are linearly fitted to obtain a fourth linear regression equation with picture order as an independent variable and Y-axis coordinates of the second vertex of the target frame of the first target as a dependent variable; the first linear regression equation, the second linear regression equation, the third linear regression equation and the fourth linear regression equation are all monomial linear equations; The second determining module is configured to determine predicted X-axis coordinates and predicted Y-axis coordinates of the first vertex of the target frame of the first target in the first picture and predicted X-axis coordinates and Y-axis coordinates of the second vertex of the target frame of the first target in the first picture through the first linear regression equation, the second linear regression equation, the third linear regression equation and the fourth linear regression equation respectively according to the order of the first picture in the picture sequence; The labeling module is configured to highlight the predicted position of the labeling target in the first picture to remind a labeling personnel to label the labeling target existing in the predicted position, the predicted position including the predicted X-axis coordinates and the predicted Y-axis coordinates of the first vertex of the target frame of the first target in the first picture and the predicted X-axis coordinates and the predicted Y-axis coordinates of the second vertex of the target frame of the first target in the first picture, and the highlighting manner including highlighting display or labeling out a contour; The second triggering module is configured to not perform auxiliary labeling on the first picture if it is detected that the first target does not continuously appear in the N-1 pictures.

Images