Image annotation method and device, electronic device and storage medium

Abstract

The present application discloses an image annotation method and device, an electronic device and a storage medium. The method includes: obtaining an image to be annotated and a first scale index; the image to be annotated carries a character point label of a first person; the character point label of the first person includes a first position of the first character point; the first scale index represents a mapping between a first size and a second size; the first size is the size of a first reference object located at the first position; the second size is the size of the first reference object in the real world; when the first scale index is greater than or equal to a first threshold, constructing a pixel point neighborhood based on the first character point; the pixel point neighborhood includes a second pixel point different from the first character point; and the position of the second pixel point is used as the character point label of the first character.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image annotation method and device, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of computer vision technology, various computer vision models have emerged, including the person positioning model. Before using the person positioning model for positioning, the person positioning model needs to be trained. The annotation information of the training image is the position of the pixel points in the person area in the training image.

[0003] Currently, the positions of pixel points in the character area of ​​the training image can be marked by manual labeling to obtain character point labels, but the accuracy of the character point labels is low. Summary of the invention

[0004] The present application provides an image annotation method and device, an electronic device, and a storage medium.

[0005] In a first aspect, a method for annotating an image is provided, the method comprising:

[0006] Acquire an image to be annotated and a first scale index; the image to be annotated carries a character point label of a first character; the character point label of the first character includes a first position of the first character point; the first scale index represents a mapping between a first size and a second size; the first size is a size of a first reference object located at the first position; the second size is a size of the first reference object in the real world;

[0007] When the first scale index is greater than or equal to a first threshold, constructing a pixel point neighborhood based on the first person point; the pixel point neighborhood includes a first pixel point different from the first person point;

[0008] The position of the first pixel point is used as a character point label of the first character.

[0009] In this aspect, it is determined whether there are unlabeled pixels in the character area based on the labeled character points and the scale index of the labeled character points. When it is determined that there are unlabeled pixels in the character area, a pixel neighborhood is constructed based on the labeled character points, and the positions of the pixels in the pixel neighborhood except the labeled character points are used as the labels of the characters corresponding to the character area, thereby improving the labeling accuracy.

[0010] In combination with any embodiment of the present application, the method further includes:

[0011] Obtaining a first length; the first length is the length of the first character in the real world;

[0012] Obtaining a position of at least one character frame of the first character according to the first position, the first scale indicator, and the first length;

[0013] The position of the at least one character frame is used as a character frame label of the first character.

[0014] In combination with any implementation manner of the present application, the position of the at least one character frame includes a second position;

[0015] The obtaining, according to the first position, the first scale index and the first length, a position of at least one character frame of the first character includes:

[0016] Determine the product of the first scale index and the first length to obtain a second length of the first person in the image to be labeled;

[0017] Based on the first position and the second length, the position of the first character frame is determined as the second position; the center of the first character frame is the first character point; the maximum length of the first character frame in the y-axis direction is not less than the second length.

[0018] In combination with any implementation manner of the present application, the shape of the first character frame is a rectangle;

[0019] The step of determining the position of the first character frame according to the first position and the second length includes:

[0020] Based on the first position and the second length, the coordinates of the diagonal vertices of the first character frame are determined; the diagonal vertices include a first vertex and a second vertex; the first vertex and the second vertex are both points on a first line segment; the first line segment is the diagonal line of the first character frame.

[0021] In combination with any embodiment of the present application, the shape of the first character frame is a square; the coordinates of the first position in the pixel coordinate system of the image to be labeled are: (p, q);

[0022] The step of determining the coordinates of the diagonal vertices of the first character frame according to the first position and the second length includes:

[0023] Determine the difference between the p and the third length to obtain a first abscissa, determine the difference between the q and the third length to obtain a first ordinate, determine the sum between the p and the third length to obtain a second abscissa, and determine the sum between the p and the third length to obtain a second ordinate; the third length is half of the second length;

[0024] The first horizontal coordinate is used as the horizontal coordinate of the first vertex, the first vertical coordinate is used as the vertical coordinate of the first vertex, the second horizontal coordinate is used as the horizontal coordinate of the second vertex, and the second vertical coordinate is used as the vertical coordinate of the second vertex.

[0025] In combination with any implementation manner of the present application, the obtaining of the first scale indicator includes:

[0026] Performing object detection processing on the image to be labeled to obtain a first object frame and a second object frame;

[0027] The third length is obtained according to the length of the first object frame in the y-axis direction, and the fourth length is obtained according to the length of the second object frame in the y-axis direction; the y-axis is the vertical axis of the pixel coordinate system of the image to be labeled;

[0028] A second scale indicator is obtained according to the third length and a fifth length of the first object in the real world, and a third scale indicator is obtained according to the fourth length and a sixth length of the second object in the real world; the first object is a detection object contained in the first object frame; the second object is a detection object contained in the second object frame; the second scale indicator represents a mapping between a third size and a fourth size; the third size is a size of a second reference object located at a second scale position; the fourth size is a size of the second reference object in the real world; the second scale position is a position in the image to be labeled determined according to the position of the first object frame; the third scale indicator represents a mapping between a fifth size and a sixth size; the fifth size is a size of a third reference object located at a third scale position; the sixth size is a size of the third reference object in the real world; the third scale position is a position in the image to be labeled determined according to the position of the second object frame;

[0029] Performing curve fitting processing on the second scale index and the third scale index to obtain a scale index map of the image to be annotated; the first pixel value in the scale index map represents a mapping between a seventh dimension and an eighth dimension; the seventh dimension is a dimension of a fourth reference object located at a fourth scale position; the eighth dimension is a dimension of the fourth reference object in the real world; the first pixel value is a pixel value of a second pixel point; the fourth scale position is a position of a third pixel point in the image to be annotated; the position of the second pixel point in the scale index map is the same as the position of the third pixel point in the image to be annotated;

[0030] The first scale indicator is obtained according to the scale indicator map and the first position.

[0031] In combination with any implementation manner of the present application, the character point label of the first character belongs to the labeled character point label; the character frame label of the first character belongs to the labeled character frame label; and the method further includes:

[0032] Get the network to be trained;

[0033] Processing the image to be annotated using the network to be trained to obtain the position of at least one person point and the position of at least one person frame;

[0034] Obtaining a first difference according to a difference between the marked character point label and the position of the at least one character point;

[0035] Obtaining a second difference according to a difference between the position of the marked character frame label and the at least one character frame;

[0036] Obtaining the loss of the network to be trained according to the first difference and the second difference;

[0037] The parameters of the network to be trained are updated based on the loss to obtain a crowd positioning network.

[0038] In combination with any embodiment of the present application, the marked character point label also includes a character point label of a second character; the character point label of the second character includes a third position of the second character point; the position of the at least one character point includes: a fourth position and a fifth position; the fourth position is the position of the character point of the first character, and the fifth position is the position of the character point of the second character;

[0039] Before obtaining the first difference based on the difference between the marked character point label and the position of the at least one character point, the method further includes:

[0040] Acquire a fourth scale indicator; the fourth scale indicator represents a mapping between a ninth dimension and a tenth dimension; the ninth dimension is a dimension of a fifth reference object located at the third position; the tenth dimension is a dimension of the fifth reference object in the real world;

[0041] The obtaining a first difference according to the difference between the marked character point label and the position of the at least one character point comprises:

[0042] obtaining a third difference according to a difference between the first position and the fourth position, and obtaining a fourth difference according to a difference between the third position and the fifth position;

[0043] According to the first scale indicator and the fourth scale indicator, a first weight of the third difference and a second weight of the fourth difference are obtained; when the first scale indicator is less than the fourth scale indicator, the first weight is greater than the second weight; when the first scale indicator is greater than the fourth scale indicator, the first weight is less than the second weight; when the first scale indicator is equal to the fourth scale indicator, the first weight is equal to the second weight;

[0044] The third difference and the fourth difference are weightedly summed according to the first weight and the second weight to obtain the first difference.

[0045] In combination with any implementation manner of the present application, the obtaining of the fourth scale indicator includes:

[0046] The fourth scale indicator is obtained according to the scale indicator map and the third position.

[0047] In combination with any embodiment of the present application, the step of processing the image to be annotated using the network to be trained to obtain the position of at least one person point and the position of at least one person frame includes:

[0048] Performing feature extraction processing on the image to be labeled to obtain first feature data;

[0049] Performing downsampling processing on the first feature data to obtain the position of the at least one character frame;

[0050] The first feature data is up-sampled to obtain the position of the at least one character point.

[0051] In combination with any implementation manner of the present application, the downsampling process is performed on the first feature data to obtain the position of the at least one character frame, including:

[0052] Downsampling the first feature data to obtain second feature data;

[0053] Performing convolution processing on the second feature data to obtain the position of the at least one character frame;

[0054] The up-sampling of the first feature data to obtain the position of the at least one character point includes:

[0055] Performing upsampling processing on the first feature data to obtain third feature data;

[0056] fusing the second feature data with the third feature data to obtain fourth feature data;

[0057] The fourth feature data is up-sampled to obtain the position of the at least one character point.

[0058] In combination with any embodiment of the present application, the method further includes:

[0059] Get the image to be processed;

[0060] The image to be processed is processed using the crowd positioning network to obtain the position of the character point of the third character and the position of the character frame of the third character; the third character is the character in the image to be processed.

[0061] In a second aspect, an image annotation device is provided, the device comprising:

[0062] An acquisition unit is used to acquire an image to be annotated and a first scale index; the image to be annotated carries a character point label of a first character; the character point label of the first character includes a first position of the first character point; the first scale index represents a mapping between a first size and a second size; the first size is a size of a first reference object located at the first position; the second size is a size of the first reference object in the real world;

[0063] A construction unit, configured to construct a pixel point neighborhood based on the first person point when the first scale index is greater than or equal to a first threshold; the pixel point neighborhood includes a first pixel point different from the first person point;

[0064] The first processing unit is used to use the position of the first pixel point as a character point label of the first character.

[0065] In combination with any embodiment of the present application, the acquisition unit is further used for:

[0066] Obtaining a first length; the first length is the length of the first character in the real world;

[0067] The device further comprises a second processing unit, wherein the second processing unit is configured to:

[0068] Obtaining a position of at least one character frame of the first character according to the first position, the first scale indicator, and the first length;

[0069] The position of the at least one character frame is used as a character frame label of the first character.

[0070] In combination with any implementation manner of the present application, the position of the at least one character frame includes a second position;

[0071] The second processing unit is used for:

[0072] Determine the product of the first scale index and the first length to obtain a second length of the first person in the image to be labeled;

[0073] Based on the first position and the second length, the position of the first character frame is determined as the second position; the center of the first character frame is the first character point; the maximum length of the first character frame in the y-axis direction is not less than the second length.

[0074] In combination with any implementation manner of the present application, the shape of the first character frame is a rectangle;

[0075] The second processing unit is used for:

[0076] Based on the first position and the second length, the coordinates of the diagonal vertices of the first character frame are determined; the diagonal vertices include a first vertex and a second vertex; the first vertex and the second vertex are both points on a first line segment; the first line segment is the diagonal line of the first character frame.

[0077] In combination with any embodiment of the present application, the shape of the first character frame is a square; the coordinates of the first position in the pixel coordinate system of the image to be labeled are: (p, q);

[0078] The second processing unit is used for:

[0079] Determine the difference between the p and the third length to obtain a first abscissa, determine the difference between the q and the third length to obtain a first ordinate, determine the sum between the p and the third length to obtain a second abscissa, and determine the sum between the p and the third length to obtain a second ordinate; the third length is half of the second length;

[0080] The first horizontal coordinate is used as the horizontal coordinate of the first vertex, the first vertical coordinate is used as the vertical coordinate of the first vertex, the second horizontal coordinate is used as the horizontal coordinate of the second vertex, and the second vertical coordinate is used as the vertical coordinate of the second vertex.

[0081] In combination with any implementation manner of the present application, the acquisition unit is used to:

[0082] Performing object detection processing on the image to be labeled to obtain a first object frame and a second object frame;

[0083] The third length is obtained according to the length of the first object frame in the y-axis direction, and the fourth length is obtained according to the length of the second object frame in the y-axis direction; the y-axis is the vertical axis of the pixel coordinate system of the image to be labeled;

[0084] A second scale indicator is obtained according to the third length and a fifth length of the first object in the real world, and a third scale indicator is obtained according to the fourth length and a sixth length of the second object in the real world; the first object is a detection object contained in the first object frame; the second object is a detection object contained in the second object frame; the second scale indicator represents a mapping between a third size and a fourth size; the third size is a size of a second reference object located at a second scale position; the fourth size is a size of the second reference object in the real world; the second scale position is a position in the image to be labeled determined according to the position of the first object frame; the third scale indicator represents a mapping between a fifth size and a sixth size; the fifth size is a size of a third reference object located at a third scale position; the sixth size is a size of the third reference object in the real world; the third scale position is a position in the image to be labeled determined according to the position of the second object frame;

[0085] Performing curve fitting processing on the second scale index and the third scale index to obtain a scale index map of the image to be annotated; the first pixel value in the scale index map represents a mapping between a seventh dimension and an eighth dimension; the seventh dimension is a dimension of a fourth reference object located at a fourth scale position; the eighth dimension is a dimension of the fourth reference object in the real world; the first pixel value is a pixel value of a second pixel point; the fourth scale position is a position of a third pixel point in the image to be annotated; the position of the second pixel point in the scale index map is the same as the position of the third pixel point in the image to be annotated;

[0086] The first scale indicator is obtained according to the scale indicator map and the first position.

[0087] In combination with any implementation manner of the present application, the character point tag of the first character belongs to the marked character point tag; the character frame tag of the first character belongs to the marked character frame tag; and the acquisition unit is further used to:

[0088] Get the network to be trained;

[0089] The device further comprises a third processing unit, wherein the third processing unit is configured to:

[0090] Processing the image to be annotated using the network to be trained to obtain the position of at least one person point and the position of at least one person frame;

[0091] Obtaining a first difference according to a difference between the marked character point label and the position of the at least one character point;

[0092] Obtaining a second difference according to a difference between the position of the marked character frame label and the at least one character frame;

[0093] Obtaining the loss of the network to be trained according to the first difference and the second difference;

[0094] The parameters of the network to be trained are updated based on the loss to obtain a crowd positioning network.

[0095] In combination with any embodiment of the present application, the marked character point label also includes a character point label of a second character; the character point label of the second character includes a third position of the second character point; the position of the at least one character point includes: a fourth position and a fifth position; the fourth position is the position of the character point of the first character, and the fifth position is the position of the character point of the second character;

[0096] The acquisition unit is further configured to acquire a fourth scale index before obtaining the first difference based on the difference between the labeled character point label and the position of the at least one character point; the fourth scale index represents a mapping between a ninth dimension and a tenth dimension; the ninth dimension is a dimension of a fifth reference object located at the third position; and the tenth dimension is a dimension of the fifth reference object in the real world;

[0097] The third processing unit is used for:

[0098] obtaining a third difference according to a difference between the first position and the fourth position, and obtaining a fourth difference according to a difference between the third position and the fifth position;

[0099] According to the first scale indicator and the fourth scale indicator, a first weight of the third difference and a second weight of the fourth difference are obtained; when the first scale indicator is less than the fourth scale indicator, the first weight is greater than the second weight; when the first scale indicator is greater than the fourth scale indicator, the first weight is less than the second weight; when the first scale indicator is equal to the fourth scale indicator, the first weight is equal to the second weight;

[0100] The third difference and the fourth difference are weightedly summed according to the first weight and the second weight to obtain the first difference.

[0101] In combination with any implementation manner of the present application, the acquisition unit is used to:

[0102] The fourth scale indicator is obtained according to the scale indicator map and the third position.

[0103] In combination with any implementation manner of the present application, the third processing unit is used for:

[0104] Performing feature extraction processing on the image to be labeled to obtain first feature data;

[0105] Performing downsampling processing on the first feature data to obtain the position of the at least one character frame;

[0106] The first feature data is up-sampled to obtain the position of the at least one character point.

[0107] In combination with any implementation manner of the present application, the third processing unit is used for:

[0108] Downsampling the first feature data to obtain second feature data;

[0109] Performing convolution processing on the second feature data to obtain the position of the at least one character frame;

[0110] The up-sampling of the first feature data to obtain the position of the at least one character point includes:

[0111] Performing upsampling processing on the first feature data to obtain third feature data;

[0112] fusing the second feature data with the third feature data to obtain fourth feature data;

[0113] The fourth feature data is up-sampled to obtain the position of the at least one character point.

[0114] In combination with any embodiment of the present application, the acquisition unit is further used for:

[0115] Get the image to be processed;

[0116] The device further includes a fourth processing unit, wherein the fourth processing unit is configured to:

[0117] The image to be processed is processed using the crowd positioning network to obtain the position of the character point of the third character and the position of the character frame of the third character; the third character is the character in the image to be processed.

[0118] According to a third aspect, a processor is provided, wherein the processor is used to execute the method according to the first aspect and any possible implementation thereof.

[0119] In a fourth aspect, an electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation method thereof.

[0120] In a fifth aspect, a computer-readable storage medium is provided, in which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the method as described in the first aspect above and any possible implementation method thereof.

[0121] In a sixth aspect, a computer program product is provided, which includes a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is enabled to execute the method of the above-mentioned first aspect and any possible implementation thereof.

[0122] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0123] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0124] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0125] Figure 1 A schematic diagram of a crowd image provided in an embodiment of the present application;

[0126] Figure 2 A schematic diagram of a pixel coordinate system provided in an embodiment of the present application;

[0127] Figure 3 A schematic diagram of a flow chart of an image annotation method provided in an embodiment of the present application;

[0128] Figure 4 A schematic diagram of an image provided in an embodiment of the present application;

[0129] Figure 5 A schematic diagram of an image to be annotated provided in an embodiment of the present application;

[0130] Figure 6 A schematic diagram of a process flow of another image annotation method provided in an embodiment of the present application;

[0131] Figure 7 A schematic diagram of a process flow of another image annotation method provided in an embodiment of the present application;

[0132] Figure 8 A schematic diagram of a sign provided in an embodiment of the present application;

[0133] Fig. 9 A schematic diagram of a process flow of another image annotation method provided in an embodiment of the present application;

[0134] Fig.10 A schematic diagram of elements in the same position provided in an embodiment of the present application;

[0135] Fig.11 A schematic diagram of the structure of a crowd positioning network provided in an embodiment of the present application;

[0136] Fig.12 A schematic diagram of the structure of a backbone network provided in an embodiment of the present application;

[0137] Fig.13 A schematic diagram of the structure of a character point branch and a character frame branch provided in an embodiment of the present application;

[0138] Fig.14 A schematic diagram of the structure of an image annotation device provided in an embodiment of the present application;

[0139] Fig.15 A schematic diagram of the hardware structure of an image annotation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0140] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0141] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0142] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0143] First, some concepts that will appear below are defined. In the embodiment of the present application, the image scale corresponding to the person who is close in the image is large, and the image scale corresponding to the person who is far away in the image is small. In the embodiment of the present application, "far" means that the distance between the real person corresponding to the person in the image and the imaging device that collects the above image is far, and "near" means that the distance between the real person corresponding to the person in the image and the imaging device that collects the above image is close.

[0144] In an image, the pixel area covered by a nearby person is larger than the pixel area covered by a distant person. For example, Figure 1 In the figure, person A is closer than person B, and the area of ​​the pixel area covered by person A is larger than the area of ​​the pixel area covered by person B. The scale of the pixel area covered by a nearby person is large, and the scale of the pixel area covered by a distant person is small. Therefore, the area of ​​the pixel area covered by a person is positively correlated with the scale of the pixel area covered by the person.

[0145] In the embodiments of the present application, the positions in the image refer to the positions in the pixel coordinates of the image. The horizontal coordinate of the pixel coordinate system in the embodiments of the present application is used to indicate the column number where the pixel point is located, and the vertical coordinate of the pixel coordinate system is used to indicate the row number where the pixel point is located. For example, in Figure 2 In the image shown, the upper left corner of the image is taken as the coordinate origin O, the direction parallel to the row of the image is the direction of the X axis, and the direction parallel to the column of the image is the direction of the Y axis, and the pixel coordinate system is constructed as XOY. The units of the horizontal and vertical coordinates are both pixels. For example, Figure 2 Pixel A in 11 The coordinates of pixel A are (1, 1). 23 The coordinates of pixel A are (3, 2). 42 The coordinates are (2, 4), pixel A 34 The coordinates are (4, 3).

[0146] In the embodiments of the present application, [a, b] represents a value interval greater than or equal to a and less than or equal to b; (c, d] represents a value interval greater than c and less than or equal to d; [e, f) represents a value interval greater than or equal to e and less than f.

[0147] The execution subject of the embodiment of the present application is an image annotation device. Optionally, the image annotation device can be one of the following: a mobile phone, a computer, a server, and a tablet computer. The embodiment of the present application is described below in conjunction with the drawings in the embodiment of the present application.

[0148] See also Figure 3 , Figure 3 It is a flowchart of an image annotation method provided in an embodiment of the present application.

[0149] 301. Obtain an image to be labeled and a first scale index.

[0150] In an embodiment of the present application, the image to be annotated may be any image. For example, the image to be annotated may include a person. The image to be annotated may include only a human head, without a torso or limbs (hereinafter, the torso and limbs are referred to as a human body). The image to be annotated may also include only a human body, without a head. The image to be annotated may also include only lower limbs or upper limbs. The present application does not limit the specific human body area contained in the image to be annotated. For another example, the image to be annotated may include animals. For another example, the image to be annotated may include plants. The present application does not limit the content contained in the image to be annotated.

[0151] In the image to be annotated, the pixel area covered by the person point can be regarded as the person area, wherein the person area is the pixel area covered by the human body. For example, the area covered by the first person point belongs to the pixel area covered by the head. For another example, the area covered by the first person point belongs to the pixel area covered by the arm. For another example, the area covered by the first person point belongs to the pixel area covered by the torso.

[0152] In the embodiment of the present application, the image to be annotated carries a character point label of the first character. The character point label of the first character includes the first position of the first character point. That is, the first position in the image to be annotated belongs to the character area where the first position is the first character.

[0153] In an embodiment of the present application, in an image, a scale indicator at a certain location (including the first scale indicator mentioned above, as well as the second scale indicator, the third scale indicator, and the fourth scale indicator to be described below) represents a mapping relationship between the size of an object at that location and the size of the object in the real world.

[0154] In one possible implementation, the scale metric at a location represents the number of pixels required to represent 1 meter in the real world at that location. Figure 4 In the image shown, pixel A 31 The scale index of the location is 50, pixel A 13 The scale index of the location is 20. Then at pixel A 31 The location represents the number of pixels required for 1 meter in the real world, which is 50.

[0155] In another possible implementation, the scale index at a certain location represents the ratio between the size of an object at that location and the size of the object in the real world. Figure 4 In the image shown, object 1 is located at pixel A 13 The location of object 2 is at pixel A 31 Location. Pixel A 31 The scale index of the location is 50, pixel A 13 The scale index of the location is 20. Then the ratio between the size of object 1 in the image and the size of object 1 in the real world is 20, and the ratio between the size of object 2 in the image and the size of object 2 in the real world is 50.

[0156] In another possible implementation, the scale indicator at a certain location represents the inverse of the ratio between the size of an object at that location and the size of the object in the real world. Figure 4 In the image shown, object 1 is located at pixel A 13 The location of object 2 is at pixel A 31 Location. Pixel A 31 The scale index of the location is 50, pixel A 13 The scale index of the location is 20. Then the ratio between the size of object 1 in the real world and the size of object 1 in the image is 20, and the ratio between the size of object 2 in the real world and the size of object 2 in the image is 50.

[0157] Optionally, positions with the same scale have the same scale index. For example, Figure 4 In the image shown, pixel A 11 The scale and pixel A 12 The scale and pixel A 13 The scales are the same, pixel A 21 The scale and pixel A 22 The scale and pixel A 23 The scales are the same, pixel A 31 The scale and pixel A 32 The scale and pixel A 33 The scales of pixels are the same. 11 The scale index, pixel A 12 The scale index, pixel A 13 The scale indicators are the same, pixel A 21 The scale index, pixel A 22 The scale index, pixel A 23 The scale indicators are the same, pixel A 31The scale index, pixel A 32 The scale index, pixel A 33 The scale indicators are the same.

[0158] In the embodiment of the present application, the first scale indicator is a scale indicator of the first position. Assuming that the first reference object is located at the first position, the first scale indicator represents a mapping between a first size and a second size, wherein the first size is the size of the first reference object in the image to be annotated, and the second size is the size of the first reference object in the real world.

[0159] In one implementation of acquiring an image to be annotated, an image annotation device receives an image to be annotated input by a user through an input component, wherein the input component includes a keyboard, a mouse, a touch screen, a touch pad, an audio input device, and the like.

[0160] In another implementation of acquiring an image to be annotated, the image annotating device receives the image to be annotated sent by a first terminal. Optionally, the first terminal may be any one of the following: a mobile phone, a computer, a tablet computer, a server, or a wearable device.

[0161] In another implementation of acquiring the image to be annotated, the image annotation device may acquire the image to be annotated by acquiring the image through an imaging component. Optionally, the imaging component may be a camera.

[0162] In an implementation of obtaining the first scale indicator, the image annotation apparatus receives the first scale indicator input by the user through an input component. The input component includes: a keyboard, a mouse, a touch screen, a touch pad, an audio input device, and the like.

[0163] In another implementation of obtaining the first scale indicator, the image annotation device receives the first scale indicator sent by the second terminal. Optionally, the second terminal can be any one of the following: a mobile phone, a computer, a tablet computer, a server, a wearable device. The second terminal can be the same as the first terminal or different.

[0164] 302. When the first scale index is greater than or equal to a first threshold, construct a pixel point neighborhood based on the first person point.

[0165] In traditional image annotation methods, the positions of the pixels contained in the person area in the image to be annotated are annotated manually to obtain person point labels. Since there may be a large person area in the image to be annotated, the person point labels obtained by traditional methods (such as the person point labels carried by the image to be annotated) may not completely cover the entire person area.

[0166] Considering that in the image to be annotated, the farther away from the x-axis of the pixel coordinate system, the larger the area of ​​the person region, and the scale index of a certain position in the image to be annotated can be used to characterize the distance between the position and the x-axis. The image annotation device determines the distance between the person region and the x-axis based on the scale index, and then determines whether there are unannotated pixels in the person region.

[0167] Since the scale index of a position in the image to be annotated is positively correlated with the distance between the position and the x-axis, the image annotation device determines whether there are unannotated pixels in the person area at the position based on whether the scale index is greater than or equal to the first threshold.

[0168] In a possible implementation, the first scale index is greater than or equal to the first threshold, indicating that there are unlabeled pixels in the character region of the first character. Optionally, the specific size of the first threshold can be determined according to actual needs. Optionally, the first threshold is 16.

[0169] Because the unlabeled pixels in the human region are usually close to the boundary of the human region, and the labeled pixels in the human region are usually close to the center of the human region, when it is determined that there are unlabeled pixels in the human region, the image labeling device can construct a pixel neighborhood based on the labeled pixels, so that the pixel neighborhood includes pixels other than the labeled pixels, and label the pixels other than the labeled pixels.

[0170] In one possible implementation, when the first scale index is greater than or equal to a first threshold, the image annotation device constructs a pixel neighborhood based on the first person point, and the pixel neighborhood includes at least one pixel point (such as the first pixel point) different from the first person point.

[0171] In the embodiment of the present application, the method of constructing the pixel neighborhood is not limited. Figure 5 In the image to be labeled shown, the first person point is pixel A 32 The image annotation device can be used to 32 The distance between the pixels is 1 pixel, and the pixel neighborhood is constructed as the pixel point in the pixel neighborhood. At this time, the pixel neighborhood includes: pixel point A 21 , pixel A 22 , pixel A 23 , pixel A 31 , pixel A 32 , pixel A 33 , pixel A 41 , pixel A 42 , pixel A 43 .

[0172] The image annotation device can also construct a pixel point neighborhood with a size of 2*2 based on the first person point. In this case, the pixel point neighborhood includes: pixel point A 21 , pixel A 22 , pixel A 31 , pixel A 32 .

[0173] The image annotation device can also use pixel point A 32 The pixel neighborhood is constructed with the center of the circle and the radius of 1.5 pixels. At this time, the pixel neighborhood includes: pixel A 21 Part of the area, pixel A 22 , pixel A 23 Part of the area, pixel A 31 , pixel A 32 , pixel A 33 , pixel A 41 Part of the area, pixel A 42 , pixel A 43 parts of the area.

[0174] Since the area of ​​the character region is larger, the number of unlabeled pixels in the character region may be greater. As an optional implementation, when the first scale index is at [first threshold, second threshold), the pixel point that is 1 pixel away from the first character point is used as the pixel point in the pixel neighborhood to construct the pixel neighborhood; when the first scale index is greater than or equal to the second threshold, the pixel point that is 2 pixels away from the first character point is used as the pixel point in the pixel neighborhood to construct the pixel neighborhood.

[0175] 303. Use the position of the first pixel point as a character point label of the first character.

[0176] After constructing a pixel point neighborhood based on the first person point, the image annotation device may annotate the first pixel point, that is, use the position of the first pixel point as the person point label of the first person.

[0177] Optionally, the image labeling device may label all pixel points in the pixel point neighborhood except the first person point, that is, use the positions of all pixel points in the pixel point neighborhood except the first person point as the person point label of the first person.

[0178] In the embodiment of the present application, the labeled person points and the scale index of the labeled person points are used to determine whether there are unlabeled pixels in the person area. When it is determined that there are unlabeled pixels in the person area, a pixel neighborhood is constructed based on the labeled person points, and the positions of the pixels in the pixel neighborhood except the labeled person points are used as the labels of the person corresponding to the person area, thereby improving the labeling accuracy.

[0179] See also Figure 6 , Figure 6 It is a flowchart of another image annotation method provided in an embodiment of the present application.

[0180] 601. Obtain a first length.

[0181] In the embodiment of the present application, the first length is the length of the first character in the real world. For example, the first length may be the height of the first character in the real world. For another example, the first length may be the length of the face of the first character in the real world. For another example, the first length may be the length of the head of the first character in the real world.

[0182] In an implementation of obtaining the first length, the image annotation apparatus receives the first length input by the user through an input component. The input component includes: a keyboard, a mouse, a touch screen, a touch pad, an audio input device, and the like.

[0183] In another implementation of obtaining the first length, the image annotation device receives the first length sent by a third terminal. Optionally, the third terminal can be any one of the following: a mobile phone, a computer, a tablet computer, a server, a wearable device. The third terminal can be the same as or different from the first terminal.

[0184] 602. Obtain a position of at least one character frame of the first character according to the first position, the first scale index, and the first length.

[0185] In the embodiment of the present application, the pixel point area included in the character frame can be regarded as the character area. For example, the character frame of the first character includes the character area of ​​the first character.

[0186] In the embodiment of the present application, the character frame can be of any shape, and the present application does not limit the shape of the character frame. Optionally, the shape of the character frame includes at least one of the following: rectangle, diamond, circle, ellipse, polygon.

[0187] In an embodiment of the present application, the form of expression of the position of the character frame in the image to be annotated may be determined according to the shape of the character frame. For example, when the shape of the character frame is a rectangle, the position of the character frame may include the coordinates of any pair of diagonal vertices in the character frame, wherein a pair of diagonal vertices refers to two vertices on the diagonal line of the character frame. For another example, when the shape of the character frame is a rectangle, the position of the character frame may include: the position of the geometric center of the character frame, the length of the character frame, and the width of the character frame. For another example, when the shape of the character frame is a circle, the position of the character frame may include: the center of the character frame and the radius of the character frame.

[0188] According to the first position, the first scale index and the first length, the position of at least one character frame of the first character can be obtained. Taking obtaining the first character frame as an example, the implementation process of obtaining the position of the character frame according to the first position, the first scale index and the first length is described in detail below.

[0189] In a possible implementation, the product of the first scale index and the first length is calculated to obtain the second length of the first person in the image to be annotated. According to the first position and the second length, the position of the first person frame can be determined as the second position, wherein the center of the first person frame is the first person point, and the maximum length of the first person frame in the y-axis direction is not less than the second length.

[0190] In the embodiment of the present application, the y-axis is the vertical axis of the pixel coordinate system of the image to be annotated. The meaning of the maximum length in the y-axis direction can be seen in the following example. For example, the rectangular frame abcd is the character frame 1, where the coordinates of a are (4, 8), the coordinates of b are (6, 8), the coordinates of c are (6, 12), and the coordinates of d are (4, 12). At this time, the length of the character frame 1 in the y-axis direction is 12-8=4.

[0191] In an implementation of determining the position of the first character frame, the coordinates of the diagonal vertices of the first character frame are determined according to the first position and the second length, and the coordinates of the diagonal vertices are used as the position of the first character frame.

[0192] In the embodiment of the present application, the diagonal vertices include a first vertex and a second vertex, wherein the first vertex and the second vertex are two vertices on any diagonal line of the first character frame. For example, the diagonal line of the first character frame includes a first line segment, and the diagonal vertices include a first vertex and a second vertex. The first vertex and the second vertex are both points on the first line segment.

[0193] Optionally, assume that the coordinates of the first position in the pixel coordinate system of the image to be annotated are: (p, q). Calculate half of the second length to obtain the third length. Determine the difference between p and the third length to obtain the first horizontal coordinate, determine the difference between q and the third length to obtain the first vertical coordinate, determine the sum between p and the third length to obtain the second horizontal coordinate, and determine the sum between p and the third length to obtain the second vertical coordinate.

[0194] The first abscissa is used as the abscissa of the first vertex, the first ordinate is used as the ordinate of the first vertex, the second abscissa is used as the abscissa of the second vertex, and the second ordinate is used as the ordinate of the second vertex.

[0195] For example, p=20, q=18, that is, the coordinates of the first position are (20, 18). Assume that the second length is 20, that is, the third length is 10. Then the first horizontal coordinate is 20-10=10, the first vertical coordinate is 18-10=8, the second horizontal coordinate is 20+10=30, and the second vertical coordinate is 18+10=18. Then the coordinates of the first vertex are (10, 8), and the coordinates of the second vertex are (30, 18).

[0196] Optionally, assume that the coordinates of the first position in the pixel coordinate system of the image to be annotated are: (p, q). Calculate half of the second length to obtain a third length. Determine the sum of p and the third length to obtain a third horizontal coordinate, determine the difference between q and the third length to obtain a third vertical coordinate, determine the difference between p and the third length to obtain a fourth horizontal coordinate, and determine the sum of p and the third length to obtain a fourth vertical coordinate.

[0197] The third abscissa is used as the abscissa of the first vertex, the third ordinate is used as the ordinate of the first vertex, the fourth abscissa is used as the abscissa of the second vertex, and the fourth ordinate is used as the ordinate of the second vertex.

[0198] For example, p=20, q=18, that is, the coordinates of the first position are (20, 18). Assume that the second length is 20, that is, the third length is 10. Then the third horizontal coordinate is 20+10=30, the third vertical coordinate is 18-10=8, the fourth horizontal coordinate is 20-10=10, and the fourth vertical coordinate is 18+10=18. Then the coordinates of the first vertex are (30, 8), and the coordinates of the second vertex are (10, 18).

[0199] In another implementation of determining the position of the first character frame, the position of the first character frame is determined as the second position according to the first position and the second length. The shape of the first character frame is a circle, the center of the first character frame is the first character point, and the diameter of the first character frame is the second length.

[0200] In another implementation of determining the position of the first character frame, the position of the first character frame is determined as the second position based on the first position and the second length. The shape of the first character frame is a rectangle, the center of the first character frame is the first character point, the length of the first character frame is the product of the first value and the second length, and the width of the first character frame is the product of the second value and the second length. Optionally, the first value is 1 and the second value is 1 / 4.

[0201] 603. Use the position of the at least one character frame as a character frame label of the first character.

[0202] In the embodiment of the present application, the position of the character frame is obtained by using the marked character points and the scale index of the marked character points. The position of the character frame is used as the label of the corresponding character, thereby marking the character frame label of the image to be marked.

[0203] See also Figure 7 , Figure 7 It is a flowchart of a possible method for obtaining a first scale indicator provided in an embodiment of the present application.

[0204] 701. Perform object detection processing on the image to be labeled to obtain a first object frame and a second object frame.

[0205] In the embodiment of the present application, the length of the object to be detected in the object detection process in the real world is close to a certain value. For example, the average length of a human face is 20 centimeters, and the object to be detected in the object detection process can be a human face. For another example, the average height of a person is 1.65 meters, and the object to be detected in the object detection process can be a human body. For another example, in the waiting room, if Figure 8 The heights of the signboards shown are all determined (eg, 2.5 meters), and the detection object of the object detection process may be the signboard. Optionally, the object detection process is a face detection process.

[0206] In one possible implementation, object detection processing for the labeled image can be implemented by a convolutional neural network. The convolutional neural network is trained by using the image with labeled information as training data, so that the trained convolutional neural network can complete the object detection processing for the image. The labeled information of the image in the training data is the position information of the object box, and the object box contains the detection object of the object detection processing.

[0207] In another possible implementation method, object detection processing can be implemented through a person detection algorithm, wherein the person detection algorithm can be one of the following: you only look once (YOLO), deformable part model (DMP), single shot multiBox detector (SSD), Faster-RCNN algorithm, etc. This application does not limit the person detection algorithm for implementing object detection processing.

[0208] In the embodiment of the present application, the detection object contained in the first object frame is different from the detection object contained in the second object frame. For example, the detection object contained in the first object frame is Zhang San's face, and the detection object contained in the second object frame is Li Si's face. For another example, the detection object contained in the first object frame is Zhang San's face, and the detection object contained in the second object frame is a signboard.

[0209] 702. Obtain a third length according to the length of the first object frame in the y-axis direction, and obtain a fourth length according to the length of the second object frame in the y-axis direction.

[0210] The image annotation device can obtain the length of the first object frame in the y-axis direction, i.e., the third length, according to the position of the first object frame. The image processing can obtain the length of the second object frame in the y-axis direction, i.e., the fourth length, according to the position of the second object frame.

[0211] 703. Obtain a second scale index according to the third length and a fifth length of the first object in the real world, and obtain a third scale index according to the fourth length and a sixth length of the second object in the real world.

[0212] In an embodiment of the present application, the second scale indicator is a scale indicator of a second scale position, wherein the second scale position is a position determined in the image to be annotated based on the position of the first object frame. Assuming that the second reference object is located at the second scale position, the second scale indicator represents a mapping between a third size and a fourth size, wherein the third size is the size of the second reference object in the image to be annotated, and the fourth size is the size of the second reference object in the real world. The third scale indicator is a scale indicator of a third scale position, wherein the third scale position is a position determined in the image to be annotated based on the position of the second object frame. Assuming that the third reference object is located at a third scale position, the third scale indicator represents a mapping between a fifth size and a sixth size, wherein the fifth size is the size of the third reference object in the image to be annotated, and the sixth size is the size of the third reference object in the real world.

[0213] In the embodiment of the present application, an object point can be determined according to the position of an object frame. For example, the shape of the object frame 1 is a rectangle. The image annotation device can determine the position of any vertex of the object frame 1 according to the position of the object frame 1, and then any vertex of the object frame 1 can be used as an object point.

[0214] For another example, the shape of the object frame 1 is a rectangle abcd. The center of the rectangle abcd is point e. The image annotation device can determine the coordinates of point e according to the position of the object frame 1, and then use point e as the object point.

[0215] For another example, the shape of the object frame 1 is a circle. The image annotation device can determine the position of any point on the circle according to the position of the object frame 1, and further can use any point on the circle as an object point.

[0216] The image annotation device determines the first object point according to the position of the first object frame. The image annotation device determines the second object point according to the position of the second object frame.

[0217] Optionally, the first object point is one of the following: a geometric center of the first object frame, a vertex of the first object frame. The second object point is one of the following: a geometric center of the second object frame, a vertex of the second object frame.

[0218] After determining the position of the first object point and the position of the second object point, the image annotation apparatus may use the position of the first object point as the second scale position and the position of the second object point as the third scale position.

[0219] In the embodiment of the present application, the first object and the second object are both detection objects of object detection processing. The first object is the detection object contained in the first object frame, and the second object is the detection object contained in the second object frame. The length of the first object in the real world is the fifth length, and the length of the second object in the real world is the sixth length. For example, the first object and the second object are both human faces, and the fifth length and the sixth length can both be 20 centimeters. For another example, the first object is a human face and the second object is a human body, the fifth length can be 20 centimeters, and the sixth length can be 170 centimeters.

[0220] Assume that the third length is l1, the fourth length is l2, the fifth length is l3, the sixth length is l4, the second scale index is i2, and the third scale index is i3.

[0221] In one possible implementation, l1, l2, l3, l4, i2, i3 satisfy the following formula:

[0222]

[0223] Wherein, k is a positive number. Optionally, k=1.

[0224] In another possible implementation, l1, l2, l3, l4, i2, i3 satisfy the following formula:

[0225]

[0226] Wherein, k is a positive number and t is a real number. Optionally, k=1 and t=0.

[0227] In another possible implementation, l1, l2, l3, l4, i2, i3 satisfy the following formula:

[0228]

[0229] Wherein, k is a positive number and t is a real number. Optionally, k=1 and t=0.

[0230] 704. Perform curve fitting processing on the second scale index and the third scale index to obtain a scale index graph of the image to be labeled.

[0231] Since the relationship between the scale and the ordinate in the image to be annotated can be regarded as a linear correlation, and the scale index is used to characterize the scale, the image annotation device can obtain a scale index map of the image to be annotated by performing curve fitting processing on the second scale index and the third scale index. The scale index map includes the scale index of the position of any pixel point in the image to be annotated.

[0232] Take the second pixel in the scale index map as an example. Assume that the pixel value of the second pixel (i.e., the first pixel value) is 40, and the position of the second pixel in the scale index map is the same as the position of the third pixel in the image to be annotated. Then the scale index of the position of the third pixel in the image to be annotated (i.e., the fourth scale position) is the first pixel value. Assuming that the fourth reference object is located at the fourth scale position, the first pixel value represents the mapping between the seventh size and the eighth size, wherein the seventh size is the size of the fourth reference object located at the fourth scale position, and the eighth size is the size of the fourth reference object in the real world.

[0233] 705. Obtain the first scale index according to the scale index diagram and the first position.

[0234] As described in step 704, the scale index map includes the scale index of the position of any pixel point in the image to be annotated. Therefore, according to the scale index map and the first position, the scale index of the first person point, that is, the first scale index, can be determined.

[0235] In the embodiment of the present application, the second scale index is obtained according to the third length and the fifth length, and the third scale index is obtained according to the fourth length and the sixth length. The scale index map is obtained by performing curve fitting processing on the second scale index and the third scale index, and then the scale index of the position of any pixel point in the image to be annotated can be determined according to the scale index map.

[0236] As an optional implementation, the person point (including: the first person point) in the embodiment of the present application can be a head point, and the person frame (including: the first person frame) can be a head frame. The pixel point area covered by the head point and the pixel point area included in the head frame are both head areas.

[0237] As an optional implementation, after the image annotation device obtains the character frame label based on the labeled character point label. A method for training a neural network can be performed using the image to be annotated as training data. The executor of the training method may be an image annotation device or may not be an annotation device. The embodiment of the present application does not limit the executor of the training method. For ease of expression, the executor of the training process is referred to as a training device below. Optionally, the training device may be any of the following: a mobile phone, a computer, a tablet computer, a server, or a processor.

[0238] See also Fig. 9 , Fig. 9 It is a flowchart of a neural network training method provided in an embodiment of the present application.

[0239] 901. Obtain a network to be trained.

[0240] In the embodiment of the present application, the network to be trained is any neural network. For example, the network to be trained may be composed of a stack of at least one network layer of a convolutional layer, a pooling layer, a normalization layer, a fully connected layer, a downsampling layer, and an upsampling layer. The structure of the network to be trained is not limited in the embodiment of the present application.

[0241] In one implementation of obtaining a network to be trained, a training device receives the network to be trained input by a user through an input component, wherein the input component includes a keyboard, a mouse, a touch screen, a touch pad, an audio input device, and the like.

[0242] In another implementation of obtaining the network to be trained, the training device receives the network to be trained sent by the fourth terminal. Optionally, the fourth terminal can be any of the following: a mobile phone, a computer, a tablet computer, a server, a wearable device. The fourth terminal can be the same as the first terminal or different, and this application does not limit this.

[0243] In yet another implementation of obtaining the network to be trained, the training device may obtain the pre-stored network to be trained from its own storage component.

[0244] 902. Use the network to be trained to process the image to be labeled to obtain the position of the at least one person point and the position of the at least one person frame.

[0245] The training device uses the network to be trained to process the image to be labeled containing at least one person, and can obtain the position of at least one person point of each person and the position of at least one person frame of each person.

[0246] In a possible implementation, the neural network to be trained performs feature extraction processing on the image to be annotated to obtain first feature data, performs downsampling processing on the first feature data to obtain the position of at least one character frame, and performs upsampling processing on the first feature data to obtain the position of at least one character point.

[0247] In the embodiments of the present application, the feature extraction processing may be a convolution processing, a pooling processing, or a combination of a convolution processing and a pooling processing. The present application does not limit the implementation method of the feature extraction processing.

[0248] Optionally, the image to be annotated is subjected to convolution processing step by step through multiple convolution layers in sequence to achieve feature extraction processing of the image to be annotated, and obtain first feature data carrying semantic information of the image to be annotated.

[0249] Optionally, the downsampling process includes one or more of the following combinations: convolution process, pooling process. For example, the downsampling process is convolution process. For another example, the downsampling process may be pooling process. For another example, the downsampling process may be convolution process and pooling process.

[0250] Optionally, the upsampling process includes at least one of the following processes: bilinear interpolation, nearest neighbor interpolation, high-order interpolation, and deconvolution.

[0251] As an optional implementation, the training device may perform downsampling processing on the first feature data to obtain the position of at least one character frame by performing the following steps:

[0252] 1. Downsample the first feature data to obtain the second feature data.

[0253] The training device can extract semantic information (i.e., semantic information of the image to be annotated) from the first feature data while reducing the size of the first feature data by downsampling the first feature data to obtain the second feature data.

[0254] 2. Perform convolution processing on the second feature data to obtain the position of at least one character frame.

[0255] The training device can obtain the position of at least one character frame by performing convolution processing on the second feature data and utilizing the semantic information carried in the second feature data.

[0256] In the case where the position of at least one character frame is obtained by executing step 1 and step 2, the training device can perform upsampling processing on the first feature data to obtain the position of at least one character frame by executing the following steps:

[0257] 3. Upsample the first feature data to obtain third feature data.

[0258] Since the distance between people in the image to be annotated may be very small, and the image annotation device extracts the first feature data while reducing the size of the image to be annotated by performing feature extraction processing on the image to be annotated, at least two person regions may overlap in the first feature data. This will obviously reduce the accuracy of the person points obtained subsequently.

[0259] In this step, the training device performs up-sampling processing on the first feature data to enlarge the size of the first feature data, thereby reducing the probability of the overlap of at least two character regions.

[0260] 4. The second feature data and the third feature data are fused to obtain fourth feature data.

[0261] Since the character frame label of the image to be annotated carries the scale information of the image to be annotated (including the scales of different positions in the image to be annotated), when the character frame label is used based on the position of at least one character frame obtained in step 2, the second feature data will also carry the scale information of the image to be annotated.

[0262] The training device can enrich the scale information in the third feature data and obtain the fourth feature data by fusing the second feature data with the third feature data.

[0263] As an optional implementation, when the size of the second feature data is smaller than the size of the third feature data, the training device uses the network to be trained to upsample the second feature data to obtain fifth feature data having the same size as the third feature data. The fifth feature data is fused with the third feature data to obtain fourth feature data.

[0264] Optionally, the fusion process may be one of the following: concatination in the channel dimension, and summing the elements at the same position.

[0265] In the embodiment of the present application, the elements at the same position in two data can refer to the following example. Fig.10 As shown, element A 11 The position in data A and element B 11 The element A has the same position in data B. 12 The position of element k in data A is the same as the position of element k in data B 12 The position of element A is the same as 13 The position in data A and element B 13 The element A has the same position in data B. 21 The position in data A and element B 21 The element A has the same position in data B. 22 The position in data A and element B 22 The element A has the same position in data B. 23 The position in data A and element B 23 The element A has the same position in data B. 31 The position in data A and element B 31 The element A has the same position in data B. 32 The position in data A and element B 32 The element A has the same position in data B. 33 The position in data A and element B 33 The positions in data B are the same.

[0266] 5. Upsample the fourth feature data to obtain the position of at least one character point.

[0267] The training device can obtain the position of at least one character point by using the semantic information carried in the fourth feature data by upsampling the fourth feature data.

[0268] Since the fourth feature data carries the scale information of the image to be annotated, the position of at least one character point is obtained by upsampling the fourth feature data, which can improve the accuracy of the position of at least one character point.

[0269] 903. Obtain a first difference according to a difference between the marked character point label and the position of the at least one character point.

[0270] Optionally, the first difference may be obtained by substituting the labeled character point label and the position of at least one character point into a binary cross entropy loss function.

[0271] For example, the annotated character point labels include the position of character point a and the position of character point b. At least one character point includes the position of character point c and the position of character point d. Character point a and character point c are both character points of the first character, and character point b and character point d are both character points of the second character. Substitute the position of character point a and the position of character point c into the binary cross entropy function to obtain difference A. Substitute the position of character point b and the position of character point d into the binary cross entropy function to obtain difference B. At this time, the first difference can be difference A, the first difference can also be difference B, and the first difference can also be the sum of difference A and difference B.

[0272] As an optional implementation, before the image annotation apparatus executes step 903, the following steps may be executed:

[0273] 6. Obtain the fourth scale indicator.

[0274] In the embodiment of the present application, the annotated person point label of the image to be annotated also includes a person point label of the second person. The person point label of the second person includes the third position of the second person point.

[0275] In the embodiment of the present application, the fourth scale indicator is the scale indicator of the third position. Assuming that the fifth reference object is located at the third position, the fourth scale indicator represents the mapping between the ninth size and the tenth size, wherein the ninth size is the size of the fifth reference object in the image to be annotated, and the tenth size is the size of the fifth reference object in the real world.

[0276] In an implementation of obtaining the fourth scale indicator, the image annotation apparatus receives the first scale indicator input by the user through an input component. The input component includes: a keyboard, a mouse, a touch screen, a touch pad, an audio input device, and the like.

[0277] In another implementation of obtaining the fourth scale indicator, the image annotation device receives the first scale indicator sent by the second terminal. Optionally, the fifth terminal can be any one of the following: a mobile phone, a computer, a tablet computer, a server, and a wearable device. The fifth terminal can be the same as or different from the first terminal.

[0278] After obtaining the fourth scale indicator, the image annotation device performs the following steps during the execution of step 903:

[0279] 7. A third difference is obtained based on a difference between the first position and the fourth position, and a fourth difference is obtained based on a difference between the third position and the fifth position.

[0280] In an embodiment of the present application, the position of at least one character point obtained by the training device by executing step 902 or step 6 includes: a fourth position and a fifth position, wherein the fourth position is the position of the character point of the first character, and the fifth position is the position of the character point of the second character.

[0281] The first position is the labeled character point label of the first person, the third position is the labeled character point label of the second person. The fourth position is the character point label of the first person obtained by processing the labeled image using the network to be trained, and the fifth position is the character point label of the second person obtained by processing the labeled image using the network to be trained.

[0282] The image annotation device can obtain a third difference according to a difference between the first position and the fourth position, and can obtain a fourth difference according to a difference between the third position and the fifth position.

[0283] Optionally, substituting the first position and the fourth position into the binary cross entropy function may obtain a third difference, and substituting the third position and the fifth position into the binary cross entropy function may obtain a fourth difference.

[0284] Assume that the difference between the first position and the fourth position is d1, the third difference is d2, the difference between the third position and the fifth position is d3, and the fourth difference is d4.

[0285] In one possible implementation, d1, d2, d, and d4 satisfy the following formula:

[0286]

[0287] Wherein, u is a positive number. Optionally, u=1.

[0288] In another possible implementation, d1, d2, d, and d4 satisfy the following formula:

[0289]

[0290] Wherein, u is a positive number, r is a real number. Optionally, u=1, r=0.

[0291] In another possible implementation, l1, l2, l3, l4, i2, i3 satisfy the following formula:

[0292]

[0293] Wherein, u is a positive number, r is a real number. Optionally, q=1, m=0.

[0294] 8. According to the first scale index and the fourth scale index, obtain a first weight of the third difference and a second weight of the fourth difference.

[0295] Since the area of ​​the nearby person region is larger than that of the distant person region in the image to be annotated, the number of person points in the nearby person region is greater than that in the distant person region. Assuming that the network obtained by training the training network is the trained network, this will result in the trained network having a high accuracy in detecting nearby persons (i.e., the accuracy of the position of the nearby person points is higher than that of the distant person points).

[0296] In order to improve the accuracy of the trained network in detecting distant people, the training device determines the weight of the difference corresponding to the person point according to the scale index of the person point, so that the weight of the difference corresponding to the nearby person point is smaller than the weight of the difference with the distant person point.

[0297] In one possible implementation, when the first scale indicator is smaller than the fourth scale indicator, the first weight is greater than the second weight; when the first scale indicator is greater than the fourth scale indicator, the first weight is less than the second weight; when the first scale indicator is equal to the fourth scale indicator, the first weight is equal to the second weight.

[0298] As an optional implementation, the size of the weight is negatively correlated with the scale index of the character point. Taking the first weight and the first scale index as an example, assuming that the first weight is w1, the first scale index is i1, and the maximum pixel value in the scale index map is i max , then w1, i1, i max Satisfy the following formula:

[0299]

[0300] 9. Based on the first weight and the second weight, perform weighted summation on the third difference and the fourth difference to obtain the first difference.

[0301] Assume that the first weight is w1, the second weight is w2, the third difference is d2, the fourth difference is d4, and the first difference is d5.

[0302] In one possible implementation, w1, w2, d2, d4, and d5 satisfy the following formula:

[0303] d5=w1×d2+w2×d4+v…Formula (8)

[0304] Wherein, v is a real number. Optionally, v=0.

[0305] In another possible implementation, w1, w2, d2, d4, and d5 satisfy the following formula:

[0306] d5=f×(w1×d2+w2×d4+v)…Formula (9)

[0307] Wherein, v is a real number, and f is a positive number. Optionally, v=0, f=1.

[0308] In another possible implementation, w1, w2, d2, d4, and d5 satisfy the following formula:

[0309]

[0310] Wherein, v is a real number, and f is a positive number. Optionally, v=0, f=1.

[0311] 904. Obtain a second difference according to a difference between the position of the marked character frame label and the at least one character frame.

[0312] Optionally, the second difference may be obtained by substituting the labeled character frame label and the position of at least one character frame into a binary cross entropy function.

[0313] For example, the labeled character frame labels include the position of character frame a and the position of character frame b. At least one character frame includes the position of character frame c and the position of character frame d. Character frame a and character frame c are both character frames of the first character, and character frame b and character frame d are both character frames of the second character. Substitute the position of character frame a and the position of character frame c into the binary cross entropy function to obtain difference A. Substitute the position of character frame b and the position of character frame d into the binary cross entropy function to obtain difference B. At this time, difference A and difference B are both first differences.

[0314] 905. According to the first difference and the second difference, the loss of the network to be trained is obtained.

[0315] Assume that the first difference is d5, the second difference is d6, and the loss of the network to be trained is L.

[0316] In one possible implementation, d5, d6, and L satisfy the following formula:

[0317] L=s×(d5+d6)…Formula (11)

[0318] Wherein, s is a positive number. Optionally, s=1.

[0319] In another possible implementation, d5, d6, and L satisfy the following formula:

[0320] L=s×(d5+d6)+n…Formula (12)

[0321] Wherein, s is a positive number and n is a real number. Optionally, s=1 and n=0.

[0322] In another possible implementation, w1, w2, d2, d4, and d5 satisfy the following formula:

[0323]

[0324] Wherein, s is a positive number and n is a real number. Optionally, s=1 and n=0.

[0325] 906. Update the parameters of the network to be trained based on the loss to obtain a crowd positioning network.

[0326] Optionally, the image annotation device updates the parameters of the network to be trained by reverse gradient propagation based on the loss of the network to be trained, so as to obtain a crowd positioning network.

[0327] By processing images containing people based on the crowd localization network, the character points and character frames of each character in the image can be obtained.

[0328] As an alternative implementation, see Fig.11 , Fig.11 A schematic diagram of the structure of a crowd positioning network provided in an embodiment of the present application.

[0329] By using the crowd localization network to process the image to be labeled, the position of the character point of each character and the position of the character frame of each character in the image to be labeled can be obtained. According to the position of the character point of the character and the position of the character frame of the character, the position of the character can be determined.

[0330] like Fig.11 As shown in FIG. 1 , the crowd localization network includes a backbone network, a person frame branch, and a person point branch. Scale information can be fused between the person frame branch and the person point branch. Fig.12Shown is a schematic diagram of the structure of the backbone network, which contains 13 convolutional layers and 4 pooling layers. Fig.13 The figure shows the structural schematic diagram of the character frame branch and the character point branch, where the character frame branch contains 3 downsampling layers and 1 convolution layer, and the character point branch contains 3 upsampling layers.

[0331] The first feature data can be obtained by processing the image to be annotated by the backbone network. The implementation method of the processing process can refer to the implementation method of "the trained neural network performs feature extraction processing on the annotated image to obtain the first feature data". The first feature data can be processed by the character frame branch to obtain the position of at least one character frame. The processing process can refer to steps 1 and 2. The first feature data can be processed by the character point branch to obtain the position of at least one character point. The processing process can refer to steps 3, 4 and 5, where step 4 is Fig.11 The “scale information fusion” shown.

[0332] As an optional implementation, the crowd positioning network obtained based on the technical solution provided in the present application can be used to process the image to obtain the positions of the character points and the character frames, and then the positions of the characters in the image can be determined based on the positions of the character points and the character frames.

[0333] It should be understood that the subject that executes the image processing using the crowd positioning network can be an image annotation device, a training device, or a device different from the image annotation device and the training device. For the convenience of expression, the subject that executes the image processing using the crowd positioning network is referred to as an image processing device. Optionally, the image processing device can be any of the following: a mobile phone, a computer, a tablet computer, a server, a processor.

[0334] In a possible implementation, an image processing device obtains an image to be processed, and processes the image to be processed using a crowd positioning network to obtain the position of a character point of a third person and the position of a character frame of the third person, wherein the third person is a person in the image to be processed. Then, the position of the third person in the image to be processed can be determined based on the position of the character point of the third person, or the position of the third person in the image to be processed can be determined based on the position of the character frame of the third person, or the position of the third person in the image to be processed can be determined based on the position of the character point of the third person and the position of the character frame of the third person.

[0335] For example, the position of the character point of the third character is (9, 10), the shape of the character frame of the third character is a rectangle, and the position of the character frame of the third character includes the coordinates of a pair of diagonal vertices of the rectangle: (6, 8), (12, 14). The position of the character point of the third character is used as the position of the third character in the image to be processed, and the position of the third character in the image to be processed is determined to be (9, 10). The position of the character frame of the third character is used as the position of the third character in the image to be processed, and the pixel point area contained by the rectangular character frame in the image to be processed is determined to be the pixel point area covered by the third character, wherein the coordinates of the four vertices of the rectangular character frame are: (6, 8), (6, 14), (12, 14), (12, 8), respectively.

[0336] As an optional implementation, the person point in the embodiment of the present application (including: the second person point, at least one person point in step 902, and the person point of the third person) can be a head point, and the person frame (including: at least one person frame in step 902, and the person frame of the third person) can be a head frame. The pixel point area covered by the head point and the pixel point area included in the head frame are both head areas.

[0337] Based on the technical solution provided in this application, the embodiment of this application also provides a possible application scenario.

[0338] The image annotation device uses a face detection data set to train a detection convolutional neural network (which can be any convolutional neural network) to obtain a face detection network. The images in the face detection data set all carry annotation information, and the annotation information includes the position of the face frame. Optionally, the face data set is WiderFace.

[0339] The image annotation device processes the crowd data set using a face detection network to obtain a face detection result of each image in the crowd data set and a confidence level of each face detection result. Each image in the crowd data set contains at least one head, and each image contains at least one head point label. Optionally, the face detection result with a confidence level higher than a third threshold is used as the first intermediate result. Optionally, the third threshold is 0.7.

[0340] The image annotation device obtains the length of the human face in the real world (such as 20 centimeters), and obtains a scale index map of each image in the crowd data set based on the length and the first intermediate result.

[0341] Based on the technical solution, crowd data set and scale index map of each image in the crowd data set provided in the present application, the image annotation device can annotate the head point label and head frame label of each image in the crowd data set to obtain the annotated crowd data set.

[0342] The image annotation device uses the annotated crowd data set to train the second detection network (the network structure can refer to the network structure of the crowd positioning network) to obtain a positioning network. The positioning network can be used to detect the position of the head point of each head in the image and the position of the head frame of each head.

[0343] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0344] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0345] See also Fig.14 , Fig.14 The schematic diagram of the structure of an image annotation device provided in an embodiment of the present application includes: an acquisition unit 11, a construction unit 12, a first processing unit 13, a second processing unit 14, a third processing unit 15, and a fourth processing unit 16. Among them:

[0346] The acquisition unit 11 is used to acquire an image to be annotated and a first scale index; the image to be annotated carries a character point label of a first character; the character point label of the first character includes a first position of the first character point; the first scale index represents a mapping between a first size and a second size; the first size is a size of a first reference object located at the first position; the second size is a size of the first reference object in the real world;

[0347] A construction unit 12 is used to construct a pixel point neighborhood based on the first person point when the first scale index is greater than or equal to a first threshold; the pixel point neighborhood includes a first pixel point different from the first person point;

[0348] The first processing unit 13 is configured to use the position of the first pixel point as a character point label of the first character.

[0349] In combination with any embodiment of the present application, the acquisition unit 11 is further used for:

[0350] Obtaining a first length; the first length is the length of the first character in the real world;

[0351] The device further comprises a second processing unit, wherein the second processing unit 14 is configured to:

[0352] Obtaining a position of at least one character frame of the first character according to the first position, the first scale indicator, and the first length;

[0353] The position of the at least one character frame is used as a character frame label of the first character.

[0354] In combination with any implementation manner of the present application, the position of the at least one character frame includes a second position;

[0355] The second processing unit 14 is used for:

[0356] Determine the product of the first scale index and the first length to obtain a second length of the first person in the image to be labeled;

[0357] Based on the first position and the second length, the position of the first character frame is determined as the second position; the center of the first character frame is the first character point; the maximum length of the first character frame in the y-axis direction is not less than the second length.

[0358] In combination with any implementation manner of the present application, the shape of the first character frame is a rectangle;

[0359] The second processing unit 14 is used for:

[0360] Based on the first position and the second length, the coordinates of the diagonal vertices of the first character frame are determined; the diagonal vertices include a first vertex and a second vertex; the first vertex and the second vertex are both points on a first line segment; the first line segment is the diagonal line of the first character frame.

[0361] In combination with any embodiment of the present application, the shape of the first character frame is a square; the coordinates of the first position in the pixel coordinate system of the image to be labeled are: (p, q);

[0362] The second processing unit 14 is used for:

[0363] Determine the difference between the p and the third length to obtain a first abscissa, determine the difference between the q and the third length to obtain a first ordinate, determine the sum between the p and the third length to obtain a second abscissa, and determine the sum between the p and the third length to obtain a second ordinate; the third length is half of the second length;

[0364] The first horizontal coordinate is used as the horizontal coordinate of the first vertex, the first vertical coordinate is used as the vertical coordinate of the first vertex, the second horizontal coordinate is used as the horizontal coordinate of the second vertex, and the second vertical coordinate is used as the vertical coordinate of the second vertex.

[0365] In combination with any embodiment of the present application, the acquisition unit 11 is used for:

[0366] Performing object detection processing on the image to be labeled to obtain a first object frame and a second object frame;

[0367] The third length is obtained according to the length of the first object frame in the y-axis direction, and the fourth length is obtained according to the length of the second object frame in the y-axis direction; the y-axis is the vertical axis of the pixel coordinate system of the image to be labeled;

[0368] A second scale indicator is obtained according to the third length and a fifth length of the first object in the real world, and a third scale indicator is obtained according to the fourth length and a sixth length of the second object in the real world; the first object is a detection object contained in the first object frame; the second object is a detection object contained in the second object frame; the second scale indicator represents a mapping between a third size and a fourth size; the third size is a size of a second reference object located at a second scale position; the fourth size is a size of the second reference object in the real world; the second scale position is a position in the image to be labeled determined according to the position of the first object frame; the third scale indicator represents a mapping between a fifth size and a sixth size; the fifth size is a size of a third reference object located at a third scale position; the sixth size is a size of the third reference object in the real world; the third scale position is a position in the image to be labeled determined according to the position of the second object frame;

[0369] Performing curve fitting processing on the second scale index and the third scale index to obtain a scale index map of the image to be annotated; the first pixel value in the scale index map represents a mapping between a seventh dimension and an eighth dimension; the seventh dimension is a dimension of a fourth reference object located at a fourth scale position; the eighth dimension is a dimension of the fourth reference object in the real world; the first pixel value is a pixel value of a second pixel point; the fourth scale position is a position of a third pixel point in the image to be annotated; the position of the second pixel point in the scale index map is the same as the position of the third pixel point in the image to be annotated;

[0370] The first scale indicator is obtained according to the scale indicator map and the first position.

[0371] In combination with any embodiment of the present application, the character point label of the first character belongs to the labeled character point label; the character frame label of the first character belongs to the labeled character frame label; the acquisition unit 11 is further used to:

[0372] Get the network to be trained;

[0373] The device further comprises a third processing unit 15, wherein the third processing unit 15 is configured to:

[0374] Processing the image to be annotated using the network to be trained to obtain the position of at least one person point and the position of at least one person frame;

[0375] Obtaining a first difference according to a difference between the marked character point label and the position of the at least one character point;

[0376] Obtaining a second difference according to a difference between the position of the marked character frame label and the at least one character frame;

[0377] Obtaining the loss of the network to be trained according to the first difference and the second difference;

[0378] The parameters of the network to be trained are updated based on the loss to obtain a crowd positioning network.

[0379] In combination with any embodiment of the present application, the marked character point label also includes a character point label of a second character; the character point label of the second character includes a third position of the second character point; the position of the at least one character point includes: a fourth position and a fifth position; the fourth position is the position of the character point of the first character, and the fifth position is the position of the character point of the second character;

[0380] The acquisition unit 11 is further configured to acquire a fourth scale index before obtaining the first difference based on the difference between the labeled character point label and the position of the at least one character point; the fourth scale index represents a mapping between a ninth dimension and a tenth dimension; the ninth dimension is a dimension of a fifth reference object located at the third position; and the tenth dimension is a dimension of the fifth reference object in the real world;

[0381] The third processing unit 15 is used for:

[0382] obtaining a third difference according to a difference between the first position and the fourth position, and obtaining a fourth difference according to a difference between the third position and the fifth position;

[0383] According to the first scale indicator and the fourth scale indicator, a first weight of the third difference and a second weight of the fourth difference are obtained; when the first scale indicator is less than the fourth scale indicator, the first weight is greater than the second weight; when the first scale indicator is greater than the fourth scale indicator, the first weight is less than the second weight; when the first scale indicator is equal to the fourth scale indicator, the first weight is equal to the second weight;

[0384] The third difference and the fourth difference are weightedly summed according to the first weight and the second weight to obtain the first difference.

[0385] In combination with any embodiment of the present application, the acquisition unit 11 is used for:

[0386] The fourth scale indicator is obtained according to the scale indicator map and the third position.

[0387] In combination with any embodiment of the present application, the third processing unit 15 is used for:

[0388] Performing feature extraction processing on the image to be labeled to obtain first feature data;

[0389] Performing downsampling processing on the first feature data to obtain the position of the at least one character frame;

[0390] The first feature data is up-sampled to obtain the position of the at least one character point.

[0391] In combination with any embodiment of the present application, the third processing unit 15 is used for:

[0392] Downsampling the first feature data to obtain second feature data;

[0393] Performing convolution processing on the second feature data to obtain the position of the at least one character frame;

[0394] The up-sampling of the first feature data to obtain the position of the at least one character point includes:

[0395] Performing upsampling processing on the first feature data to obtain third feature data;

[0396] fusing the second feature data with the third feature data to obtain fourth feature data;

[0397] The fourth feature data is up-sampled to obtain the position of the at least one character point.

[0398] In combination with any embodiment of the present application, the acquisition unit 11 is further used for:

[0399] Get the image to be processed;

[0400] The device further comprises a fourth processing unit 16, wherein the fourth processing unit 16 is configured to:

[0401] The image to be processed is processed using the crowd positioning network to obtain the position of the character point of the third character and the position of the character frame of the third character; the third character is the character in the image to be processed.

[0402] In the embodiment of the present application, the labeled person points and the scale index of the labeled person points are used to determine whether there are unlabeled pixels in the person area. When it is determined that there are unlabeled pixels in the person area, a pixel neighborhood is constructed based on the labeled person points, and the positions of the pixels in the pixel neighborhood except the labeled person points are used as the labels of the person corresponding to the person area, thereby improving the labeling accuracy.

[0403] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0404] Fig.15 A schematic diagram of the hardware structure of an image annotation device provided in an embodiment of the present application. The image annotation device 2 includes a processor 21, a memory 22, an input device 23, and an output device 24. The processor 21, the memory 22, the input device 23, and the output device 24 are coupled via a connector, and the connector includes various interfaces, transmission lines, or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines, buses, etc.

[0405] The processor 21 may be one or more graphics processing units (GPUs). When the processor 21 is a GPU, the GPU may be a single-core GPU or a multi-core GPU. Optionally, the processor 21 may be a processor group consisting of multiple GPUs, and the multiple processors are coupled to each other via one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0406] The memory 22 can be used to store computer program instructions and various computer program codes including the program code for executing the program code of the present application. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), which is used for related instructions and data.

[0407] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0408] It can be understood that in the embodiment of the present application, the memory 22 can be used not only to store relevant instructions, but also to store relevant data. For example, the memory 22 can be used to store the image to be labeled obtained through the input device 23, or the memory 22 can also be used to store the position of the second pixel point obtained by the processor 21, etc. The embodiment of the present application does not limit the specific data stored in the memory.

[0409] Understandably, Fig.15 Only a simplified design of an image annotation device is shown. In practical applications, the image annotation device may also include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all image annotation devices that can implement the embodiments of the present application are within the scope of protection of the present application.

[0410] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0411] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that the descriptions of the various embodiments of the present application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, refer to the records of other embodiments.

[0412] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0413] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0414] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0415] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0416] A person skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. An image annotation method, characterized in that: The method comprises: Acquire an image to be annotated and a first scale index; the image to be annotated carries a character point label of a first character; the character point label of the first character includes a first position of the first character point; the first scale index represents a mapping between a first size and a second size; the first size is a size of a first reference object located at the first position; the second size is a size of the first reference object in the real world; The obtaining of the first scale index includes: performing object detection processing on the image to be labeled to obtain a first object frame and a second object frame; obtaining a third length according to the length of the first object frame in the y-axis direction, and obtaining a fourth length according to the length of the second object frame in the y-axis direction; the y-axis is the vertical axis of the pixel coordinate system of the image to be labeled; obtaining a second scale index according to the third length and a fifth length of the first object in the real world, and obtaining a third scale index according to the fourth length and a sixth length of the second object in the real world; the first object is a detection object included in the first object frame; the second object is a detection object included in the second object frame; the second scale index represents a mapping between a third size and a fourth size; the third size is a size of a second reference object located at a second scale position; the fourth size is a size of the second reference object in the real world; the second scale position is a position in the image to be labeled determined according to the position of the first object frame; the third scale index represents a mapping between a third size and a fourth size; the third size is a size of a second reference object located at a second scale position; the fourth size is a size of the second reference object in the real world; the second scale position is a position in the image to be labeled determined according to the position of the first object frame; the third scale index represents a mapping between a third size and a fourth size. The scale index represents the mapping between the fifth size and the sixth size; the fifth size is the size of the third reference object located at the third scale position; the sixth size is the size of the third reference object in the real world; the third scale position is the position in the image to be labeled determined according to the position of the second object frame; the second scale index and the third scale index are subjected to curve fitting processing to obtain a scale index map of the image to be labeled; the first pixel value in the scale index map represents the mapping between the seventh size and the eighth size; the seventh size is the size of the fourth reference object located at the fourth scale position; the eighth size is the size of the fourth reference object in the real world; the first pixel value is the pixel value of the second pixel point; the fourth scale position is the position of the third pixel point in the image to be labeled; the position of the second pixel point in the scale index map is the same as the position of the third pixel point in the image to be labeled; the first scale index is obtained according to the scale index map and the first position; When the first scale index is greater than or equal to a first threshold, constructing a pixel point neighborhood based on the first person point; the pixel point neighborhood includes a first pixel point different from the first person point; The position of the first pixel point is used as a character point label of the first character.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining a first length; the first length is the length of the first character in the real world; Obtaining a position of at least one character frame of the first character according to the first position, the first scale indicator, and the first length; The position of the at least one character frame is used as a character frame label of the first character.

3. The method according to claim 2, characterized in that The position of the at least one character frame includes a second position; The obtaining, according to the first position, the first scale index and the first length, a position of at least one character frame of the first character includes: Determine the product of the first scale index and the first length to obtain a second length of the first person in the image to be labeled; Based on the first position and the second length, the position of the first character frame is determined as the second position; the center of the first character frame is the first character point; the maximum length of the first character frame in the y-axis direction is not less than the second length.

4. The method according to claim 3, characterized in that The shape of the first character frame is a rectangle; The step of determining the position of the first character frame according to the first position and the second length includes: Based on the first position and the second length, the coordinates of the diagonal vertices of the first character frame are determined; the diagonal vertices include a first vertex and a second vertex; the first vertex and the second vertex are both points on a first line segment; the first line segment is the diagonal line of the first character frame.

5. The method according to claim 4, characterized in that The shape of the first character frame is a square; the coordinates of the first position in the pixel coordinate system of the image to be labeled are: (p, q); The step of determining the coordinates of the diagonal vertices of the first character frame according to the first position and the second length includes: Determine the difference between the p and the third length to obtain a first abscissa, determine the difference between the q and the third length to obtain a first ordinate, determine the sum between the p and the third length to obtain a second abscissa, and determine the sum between the p and the third length to obtain a second ordinate; the third length is half of the second length; The first horizontal coordinate is used as the horizontal coordinate of the first vertex, the first vertical coordinate is used as the vertical coordinate of the first vertex, the second horizontal coordinate is used as the horizontal coordinate of the second vertex, and the second vertical coordinate is used as the vertical coordinate of the second vertex.

6. The method according to claim 1, characterized in that The character point label of the first character belongs to the labeled character point label; the character frame label of the first character belongs to the labeled character frame label; the method further includes: Get the network to be trained; Processing the image to be annotated using the network to be trained to obtain the position of at least one person point and the position of at least one person frame; Obtaining a first difference according to a difference between the marked character point label and the position of the at least one character point; Obtaining a second difference according to a difference between the position of the marked character frame label and the at least one character frame; Obtaining the loss of the network to be trained according to the first difference and the second difference; The parameters of the network to be trained are updated based on the loss to obtain a crowd positioning network.

7. The method according to claim 6, characterized in that The annotated character point label also includes a character point label of a second character; the character point label of the second character includes a third position of the second character point; the position of the at least one character point includes: a fourth position and a fifth position; the fourth position is the position of the character point of the first character, and the fifth position is the position of the character point of the second character; Before obtaining the first difference based on the difference between the marked character point label and the position of the at least one character point, the method further includes: Acquire a fourth scale indicator; the fourth scale indicator represents a mapping between a ninth dimension and a tenth dimension; the ninth dimension is a dimension of a fifth reference object located at the third position; the tenth dimension is a dimension of the fifth reference object in the real world; The obtaining a first difference according to the difference between the marked character point label and the position of the at least one character point comprises: obtaining a third difference according to a difference between the first position and the fourth position, and obtaining a fourth difference according to a difference between the third position and the fifth position; According to the first scale indicator and the fourth scale indicator, a first weight of the third difference and a second weight of the fourth difference are obtained; when the first scale indicator is less than the fourth scale indicator, the first weight is greater than the second weight; when the first scale indicator is greater than the fourth scale indicator, the first weight is less than the second weight; when the first scale indicator is equal to the fourth scale indicator, the first weight is equal to the second weight; The third difference and the fourth difference are weightedly summed according to the first weight and the second weight to obtain the first difference.

8. The method according to claim 7, characterized in that The obtaining of the fourth scale indicator includes: The fourth scale indicator is obtained according to the scale indicator map and the third position.

9. The method according to any one of claims 6 to 8, characterized in that The step of using the network to be trained to process the image to be annotated to obtain the position of at least one person point and the position of at least one person frame includes: Performing feature extraction processing on the image to be labeled to obtain first feature data; Performing downsampling processing on the first feature data to obtain the position of the at least one character frame; The first feature data is up-sampled to obtain the position of the at least one character point.

10. The method according to claim 9, characterized in that The downsampling process is performed on the first feature data to obtain the position of the at least one character frame, including: Downsampling the first feature data to obtain second feature data; Performing convolution processing on the second feature data to obtain the position of the at least one character frame; The up-sampling of the first feature data to obtain the position of the at least one character point includes: Performing upsampling processing on the first feature data to obtain third feature data; fusing the second feature data with the third feature data to obtain fourth feature data; The fourth feature data is up-sampled to obtain the position of the at least one character point.

11. The method according to any one of claims 6 to 8, characterized in that The method further comprises: Get the image to be processed; The image to be processed is processed using the crowd positioning network to obtain the position of the character point of the third character and the position of the character frame of the third character; the third character is the character in the image to be processed.

12. An image annotation device, characterized in that: The device comprises: An acquisition unit is used to acquire an image to be annotated and a first scale index; the image to be annotated carries a character point label of a first character; the character point label of the first character includes a first position of the first character point; the first scale index represents a mapping between a first size and a second size; the first size is a size of a first reference object located at the first position; the second size is a size of the first reference object in the real world; The obtaining of the first scale index includes: performing object detection processing on the image to be labeled to obtain a first object frame and a second object frame; obtaining a third length according to the length of the first object frame in the y-axis direction, and obtaining a fourth length according to the length of the second object frame in the y-axis direction; the y-axis is the vertical axis of the pixel coordinate system of the image to be labeled; obtaining a second scale index according to the third length and a fifth length of the first object in the real world, and obtaining a third scale index according to the fourth length and a sixth length of the second object in the real world; the first object is a detection object included in the first object frame; the second object is a detection object included in the second object frame; the second scale index represents a mapping between a third size and a fourth size; the third size is a size of a second reference object located at a second scale position; the fourth size is a size of the second reference object in the real world; the second scale position is a position in the image to be labeled determined according to the position of the first object frame; the third scale index represents a mapping between a third size and a fourth size; the third size is a size of a second reference object located at a second scale position; the fourth size is a size of the second reference object in the real world; the second scale position is a position in the image to be labeled determined according to the position of the first object frame; the third scale index represents a mapping between a third size and a fourth size. The scale index represents the mapping between the fifth size and the sixth size; the fifth size is the size of the third reference object located at the third scale position; the sixth size is the size of the third reference object in the real world; the third scale position is the position in the image to be labeled determined according to the position of the second object frame; the second scale index and the third scale index are subjected to curve fitting processing to obtain a scale index map of the image to be labeled; the first pixel value in the scale index map represents the mapping between the seventh size and the eighth size; the seventh size is the size of the fourth reference object located at the fourth scale position; the eighth size is the size of the fourth reference object in the real world; the first pixel value is the pixel value of the second pixel point; the fourth scale position is the position of the third pixel point in the image to be labeled; the position of the second pixel point in the scale index map is the same as the position of the third pixel point in the image to be labeled; the first scale index is obtained according to the scale index map and the first position; A construction unit, configured to construct a pixel point neighborhood based on the first person point when the first scale index is greater than or equal to a first threshold; the pixel point neighborhood includes a first pixel point different from the first person point; The first processing unit is used to use the position of the first pixel point as a character point label of the first character.

13. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is enabled to execute the method according to any one of claims 1 to 11.

Images