Picture set target labeling method and device

Abstract

The invention provides a picture set target labeling method and device, and the method comprises the steps: taking a video frame picture of each first target appearing for the first time as a first video frame picture, selecting a first target region in the first video frame picture, taking the first target region as a first target template, and taking the first target template as a second target template; adding the label information of the first target area to a first label file of the first video frame picture; from the second video frame picture to the last video frame picture, selecting a search area in the mth video frame picture; and for different grid sizes, sequentially judging whether the size of the search area can be exactly divided by each grid size, and if so, carrying out grid division on the search area according to each grid size. The invention provides a picture set target labeling method and device which are used for solving the problems that in the prior art, the workload is huge and very tedious, the time and labor cost is high, and the efficiency is low.

Description

Technical Field

[0001] The present invention relates to the technical field of image data annotation, and in particular to a method and device for annotating objects in a picture set. Background Art

[0002] In the field of computer vision, object detection is a current research hotspot, among which supervised machine learning methods have become the mainstream research direction. With the continuous improvement of hardware computing power, deeper and larger networks have achieved better detection results, but their model training relies on richer and more comprehensive data set samples.

[0003] Traditional dataset production methods rely on manual labeling, which is labor-intensive and cumbersome, requiring a large amount of time and manpower costs, and has low labeling efficiency, making it difficult to complete the labeling of large target detection datasets in a short period of time. Summary of the invention

[0004] The present invention provides a method and device for labeling targets in a picture set, which are used to solve the problems existing in the prior art such as huge and cumbersome workload, high time and labor costs, and low efficiency.

[0005] In a first aspect, the present invention provides a method for labeling an object in a picture set, comprising:

[0006] Determine whether the picture set belongs to a video frame picture set or a non-video frame picture set;

[0007] If the picture set belongs to the video frame picture set, each first target in the video frame picture set is labeled according to the following steps:

[0008] S101, taking a video frame picture in which each first target appears for the first time as a first video frame picture, selecting a first target area in the first video frame picture, taking the first target area as a first target template, and adding label information of the first target area to a first label file of the first video frame picture, wherein the label information includes a position, a size, and a target type number;

[0009] S102, starting from the second video frame picture to the last video frame picture, selecting a search area in the mth (m≥2)th video frame picture, the search area coincides with the center of the first target area of ​​the m-1th video frame picture, and the size of the search area is a preset multiple of the size of the first target area of ​​the m-1th video frame picture, and the preset multiple is greater than 1;

[0010] S103, for different grid sizes, determining in turn whether the size of the search area is divisible by each grid size, if so, dividing the search area into grids according to each grid size, if not, filling the search area so that the size of the filled search area is divisible by each grid size, and then dividing the filled search area into grids according to each grid size;

[0011] S104, scaling all the grids obtained by division to the size of the first target area of ​​the m-1th video frame picture, calculating the correlation coefficient between each scaled grid and the first target template of the m-1th video frame picture, selecting the grid before scaling corresponding to the grid with the largest correlation coefficient as the first target area of ​​the mth video frame picture, using the first target area of ​​the mth video frame picture as the first target template of the mth video frame picture, and adding the label information of the first target area of ​​the mth video frame picture to the first label file of the mth video frame picture;

[0012] S105, based on the first label file of each video frame picture, draw a first target annotated area on each video frame picture, starting from the first video frame picture, determine whether the overlap between the first target annotated area of ​​the nth (n≥1)th video frame picture and the first target actual area is lower than a first preset range, if lower, take the nth video frame picture as a new first video frame picture, and execute S101-105 for the new first video frame picture and its subsequent video frame pictures;

[0013] If the picture set belongs to the non-video frame picture set, each second target in the non-video frame picture set is labeled according to the following steps:

[0014] S201, taking the non-video frame picture in which each second target appears for the first time as the first non-video frame picture, selecting a second target area in the first non-video frame picture, taking the second target area as the second target template, and adding label information of the second target area to the second label file of the first non-video frame picture;

[0015] S202, starting from the second non-video frame picture to the last non-video frame picture, determining whether the size of the kth (k≥2) non-video frame picture is divisible by the size of the second target area, if so, performing grid division on the kth non-video frame picture according to the size of the second target area, if not, padding the kth non-video frame picture so that the size of the padded kth non-video frame picture is divisible by the size of the second target area, and then performing grid division on the padded kth non-video frame picture according to the size of the second target area;

[0016] S203, calculating the correlation coefficient between each grid and the second target template, and adding the label information of all grids whose correlation coefficient is greater than the correlation threshold to the second label file of the kth non-video frame picture;

[0017] S204. Based on the second label file of each non-video frame picture, draw a second target annotation area on each non-video frame picture, and starting from the first non-video frame picture, determine whether the overlap between the second target annotation area of ​​the lth (l≥1) non-video frame picture and the second target actual area is lower than the first preset range; if so, take the lth non-video frame picture as the new first non-video frame picture, and execute S201-204 for the new first non-video frame picture and its subsequent non-video frame pictures.

[0018] The above solution solves the problems of huge and tedious workload, high time and labor costs, and low efficiency in manual labeling, and achieves efficient and accurate labeling.

[0019] Optionally, the determining whether the picture set belongs to a video frame picture set or a non-video frame picture set includes:

[0020] sorting all the pictures in the picture set based on the picture names;

[0021] For all the sorted pictures, determine in turn whether there is temporal continuity and content continuity between adjacent pictures;

[0022] If yes, the picture set belongs to the video frame picture set;

[0023] If not, the picture set belongs to the non-video frame picture set.

[0024] Optionally, filling the search area is achieved by filling a black border around the search area.

[0025] Optionally, scaling all the grids obtained by division to the size of the first target area of ​​the m-1th video frame picture is achieved by bilinear interpolation.

[0026] Optionally, the calculation formula of the correlation coefficient is as follows:

[0027]

[0028] Where r is the correlation coefficient, X i,j is the pixel value at coordinate (i, j) in the grid, is the average pixel value of all coordinates in the grid, Y i,j is the pixel value of the coordinate (i, j) in the target template, is the average pixel value of all coordinates in the target template.

[0029] Optionally, before performing S201-204 on the new first non-video frame picture and its subsequent non-video frame pictures:

[0030] Determine whether there is a situation in which missing labels exceed a second preset range in the new first non-video frame picture;

[0031] If so, the correlation threshold is decreased.

[0032] Optionally, before performing S201-204 on the new first non-video frame picture and its subsequent non-video frame pictures:

[0033] Determine whether there is a mislabeling situation exceeding a third preset range in the first non-video frame picture;

[0034] If so, the correlation threshold is increased.

[0035] In a second aspect, the present invention provides a device for labeling an object in a picture set, comprising a judgment module, a first labeling module and a second labeling module, wherein:

[0036] The judging module is used to judge whether the picture set belongs to a video frame picture set or a non-video frame picture set;

[0037] The first labeling module is used for labeling each first target in the video frame picture set according to the following steps when the result of the judging module is that the target belongs to the video frame picture set:

[0038] S101, taking a video frame picture in which each first target appears for the first time as a first video frame picture, selecting a first target area in the first video frame picture, taking the first target area as a first target template, and adding label information of the first target area to a first label file of the first video frame picture, wherein the label information includes a position, a size, and a target type number;

[0039] S102, starting from the second video frame picture to the last video frame picture, selecting a search area in the mth (m≥2)th video frame picture, the search area coincides with the center of the first target area of ​​the m-1th video frame picture, and the size of the search area is a preset multiple of the size of the first target area of ​​the m-1th video frame picture, and the preset multiple is greater than 1;

[0040] S103, for different grid sizes, determining in turn whether the size of the search area is divisible by each grid size, if so, dividing the search area into grids according to each grid size, if not, filling the search area so that the size of the filled search area is divisible by each grid size, and then dividing the filled search area into grids according to each grid size;

[0041] S104, scaling all the grids obtained by division to the size of the first target area of ​​the m-1th video frame picture, calculating the correlation coefficient between each scaled grid and the first target template of the m-1th video frame picture, selecting the grid before scaling corresponding to the grid with the largest correlation coefficient as the first target area of ​​the mth video frame picture, using the first target area of ​​the mth video frame picture as the first target template of the mth video frame picture, and adding the label information of the first target area of ​​the mth video frame picture to the first label file of the mth video frame picture;

[0042] S105, based on the first label file of each video frame picture, draw a first target annotated area on each video frame picture, starting from the first video frame picture, determine whether the overlap between the first target annotated area of ​​the nth (n≥1)th video frame picture and the first target actual area is lower than a first preset range, if lower, take the nth video frame picture as a new first video frame picture, and execute S101-105 for the new first video frame picture and its subsequent video frame pictures;

[0043] The second labeling module is used for labeling each second target in the non-video frame picture set according to the following steps when the result of the judging module is that the target belongs to the non-video frame picture set:

[0044] S201, taking the non-video frame picture in which each second target appears for the first time as the first non-video frame picture, selecting a second target area in the first non-video frame picture, taking the second target area as the second target template, and adding label information of the second target area to the second label file of the first non-video frame picture;

[0045] S202, starting from the second non-video frame picture to the last non-video frame picture, determining whether the size of the kth (k≥2) non-video frame picture is divisible by the size of the second target area, if so, performing grid division on the kth non-video frame picture according to the size of the second target area, if not, padding the kth non-video frame picture so that the size of the padded kth non-video frame picture is divisible by the size of the second target area, and then performing grid division on the padded kth non-video frame picture according to the size of the second target area;

[0046] S203, calculating the correlation coefficient between each grid and the second target template, and adding the label information of all grids whose correlation coefficient is greater than the correlation threshold to the second label file of the kth non-video frame picture;

[0047] S204. Based on the second label file of each non-video frame picture, draw a second target annotation area on each non-video frame picture, and starting from the first non-video frame picture, determine whether the overlap between the second target annotation area of ​​the lth (l≥1) non-video frame picture and the second target actual area is lower than the first preset range; if so, take the lth non-video frame picture as the new first non-video frame picture, and execute S201-204 for the new first non-video frame picture and its subsequent non-video frame pictures.

[0048] In a third aspect, the present invention provides a computing device, comprising:

[0049] A memory for storing program instructions;

[0050] The processor is used to call the program instructions stored in the memory and execute any of the above methods according to the obtained program.

[0051] In a fourth aspect, the present invention provides a computer-readable non-volatile storage medium, comprising computer-readable instructions, which enable a computer to execute any of the above methods when the computer reads and executes the computer-readable instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A schematic diagram of a flow chart of a method for labeling a target in a picture set provided by an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of the structure of a device for labeling a picture set object provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Figure 1 The process of a method for labeling a target in a picture set provided by an embodiment of the present invention is shown in detail, including:

[0057] Determine whether the picture set belongs to a video frame picture set or a non-video frame picture set.

[0058] A video frame picture set refers to continuous pictures taken from a video at a certain time interval. There is temporal continuity and content continuity between any two pictures. For example, a picture is taken every 0.1 seconds and saved in sequence. Usually, people can judge whether it belongs to a video frame picture set based on their normal thinking.

[0059] The non-video frame picture set refers to randomly arranged pictures, and there is no temporal continuity and content continuity between any two pictures.

[0060] In an example, determining whether a picture set belongs to a video frame picture set or a non-video frame picture set includes:

[0061] Sort all the images in the collection based on their names;

[0062] For all the sorted pictures, determine in turn whether there is temporal continuity and content continuity between adjacent pictures;

[0063] If yes, the picture set belongs to the video frame picture set;

[0064] If not, the picture set belongs to a non-video frame picture set.

[0065] If the picture set belongs to a video frame picture set, label each first object in the video frame picture set according to the following steps:

[0066] S101, taking the video frame picture in which each first target appears for the first time as the first video frame picture, selecting the first target area in the first video frame picture, taking the first target area as the first target template, and adding the label information of the first target area to the first label file of the first video frame picture.

[0067] It should be noted that the video frame picture where the first target first appears is not necessarily the first picture in the video frame picture set, but the video frame pictures before the video frame picture where the first target first appears are no longer taken into consideration. Therefore, the video frame picture where the first target first appears is taken as the first video frame picture for subsequent processing.

[0068] In one example, the first target area is manually selected using a rectangular frame.

[0069] Specifically, the tag information includes the location, size, and target type number.

[0070] The position of each pixel in the two-dimensional image is determined by coordinates (x, y). In an example, the position of the first target area is the coordinates of the center pixel.

[0071] The target type number can be defined by yourself, generally starting from 0 and increasing in sequence. The target type number is unique and cannot be changed after it is determined. For example: there are three types of targets in a picture: cats, dogs and people. If the targets to be detected include these three types, then the target type to be detected in the picture is considered to be 3. When the self-defined arrangement order is "cat, dog, person", the target type number of cat is 0, the target type number of dog is 1, and the target type number of person is 2; if the targets to be detected only include cats and dogs, then the target type to be detected in the picture is considered to be 2. When the self-defined arrangement order is "cat, dog", the target type number of cat is 0, and the target type number of dog is 1.

[0072] In one example, the first label file is in txt format, and the name of the first label file is the same as the name of the corresponding video frame image.

[0073] S102 . Starting from the second video frame picture to the last video frame picture, select a search area in the mth (m≥2) video frame picture.

[0074] Specifically, the search area coincides with the center of the first target area of ​​the m-1th video frame picture, and the size of the search area is a preset multiple of the size of the first target area of ​​the m-1th video frame picture, and the preset multiple is greater than 1.

[0075] In one example, the preset multiple is set to 1.5.

[0076] S103. For different grid sizes, determine in turn whether the size of the search area is divisible by each grid size. If so, divide the search area into grids according to each grid size. If not, fill the search area so that the size of the filled search area is divisible by each grid size, and then divide the filled search area into grids according to each grid size.

[0077] In one example, different grid sizes include 0.9 times, 1 times, and 1.1 times the size of the first target area of ​​the m-1th video frame. After the search area is grid-divided according to these three grid sizes, the search area can be divided into a plurality of grids of three different sizes. 0.9 times, 1 times, and 1.1 times are only general empirical values, and different grid sizes can also include grids of more other sizes.

[0078] In one example, the search area is filled by filling a black border around the search area. Since the pixel value corresponding to black is 0, it will not have a significant impact on the original image, so it is recommended to use a black border to fill.

[0079] S104. Scale all the grids obtained by division to the size of the first target area of ​​the m-1th video frame picture, calculate the correlation coefficient between each scaled grid and the first target template of the m-1th video frame picture, select the grid before scaling corresponding to the grid with the largest correlation coefficient as the first target area of ​​the mth video frame picture, use the first target area of ​​the mth video frame picture as the first target template of the mth video frame picture, and add the label information of the first target area of ​​the mth video frame picture to the first label file of the mth video frame picture.

[0080] In one example, scaling all the grids obtained by division to the size of the first target area of ​​the m-1th video frame picture is achieved by bilinear interpolation.

[0081] The next video frame image calculates the correlation coefficient based on the first target template of the previous video frame image, so that the first target template can be refreshed as the first target changes, thereby maintaining a high similarity between the first target template and the first target in the next image to improve the accuracy of labeling.

[0082] In one example, the correlation coefficient is calculated as follows:

[0083]

[0084] Among them, r is the correlation coefficient, which ranges from -1 to 1. r close to 1 indicates high correlation, and close to -1 indicates low correlation. i,j is the pixel value at coordinate (i, j) in the grid, is the average pixel value of all coordinates in the grid, Y i,j is the pixel value of the coordinate (i, j) in the target template, is the average pixel value of all coordinates in the target template.

[0085] S105. Draw a first target annotation area on each video frame image based on the first label file of each video frame image. Starting from the first video frame image, determine whether the overlap between the first target annotation area of ​​the nth (n≥1)th video frame image and the first target actual area is lower than a first preset range. If so, take the nth video frame image as the new first video frame image, and execute S101-105 for the new first video frame image and its subsequent video frame images.

[0086] In one example, the first preset range is 50%.

[0087] Usually, it can be determined whether the overlap is lower than the first preset range by manual inspection. During manual inspection, 5 images can be checked in 1 second, and during manual annotation, a simple image also takes tens of seconds. Therefore, although the inspection is done manually, the time saving is still very obvious compared to manual annotation.

[0088] Through continuous labeling and verification, the target labeling of all video frame picture sets can be completed eventually.

[0089] If the picture set belongs to a non-video frame picture set, each second target in the non-video frame picture set is labeled according to the following steps:

[0090] S201. Use the non-video frame picture where each second target appears for the first time as the first non-video frame picture, select the second target area in the first non-video frame picture, use the second target area as the second target template, and add the label information of the second target area to the second label file of the first non-video frame picture.

[0091] It should be noted that the non-video frame picture where the second target first appears is not necessarily the first picture in the non-video frame picture set, but the non-video frame picture before the non-video frame picture where the second target first appears is no longer taken into consideration. Therefore, the non-video frame picture where the second target first appears is taken as the first non-video frame picture for subsequent processing.

[0092] In one example, the second target area is manually selected using a rectangular frame.

[0093] In an example, the second label file is in txt format, and the name of the second label file is the same as the name of the corresponding non-video frame picture.

[0094] S202. Starting from the second non-video frame picture to the last non-video frame picture, determine whether the size of the kth (k≥2) non-video frame picture is divisible by the size of the second target area; if so, divide the kth non-video frame picture into a grid according to the size of the second target area; if not, fill the kth non-video frame picture so that the size of the filled kth non-video frame picture is divisible by the size of the second target area, and then divide the filled kth non-video frame picture into a grid according to the size of the second target area.

[0095] In an example, filling the kth non-video frame picture is achieved by filling a black border around the kth non-video frame picture.

[0096] S203 , calculating the correlation coefficient between each grid and the second target template, and adding the label information of all grids whose correlation coefficients are greater than the correlation threshold to the second label file of the kth non-video frame image.

[0097] In one example, the correlation threshold value ranges from 0 to 1, and a recommended value is 0.7.

[0098] It should be noted that as long as the correlation coefficient of a grid is greater than the correlation threshold, its label information will be added to the second label file. Therefore, the second label file of the non-video frame image is likely to store label information of multiple grids.

[0099] S204. Draw a second target annotation area on each non-video frame picture based on the second label file of each non-video frame picture. Starting from the first non-video frame picture, determine whether the overlap between the second target annotation area of ​​the lth (l≥1) non-video frame picture and the second target actual area is lower than a first preset range. If so, take the lth non-video frame picture as the new first non-video frame picture, and execute S201-204 for the new first non-video frame picture and its subsequent non-video frame pictures.

[0100] Through continuous labeling and verification, the target labeling of all non-video frame picture sets can be completed eventually.

[0101] In an example, before performing S201-204 for the new first non-video frame picture and its subsequent non-video frame pictures:

[0102] Determine whether there is a situation in which missing labels exceed a second preset range in the new first non-video frame image;

[0103] If present, reduce the relevance threshold.

[0104] In one example, the second preset range is 50%, that is, the number of grids whose correlation coefficients are greater than the correlation threshold is less than half of the number of actual targets.

[0105] In an example, before performing S201-204 for the new first non-video frame picture and its subsequent non-video frame pictures:

[0106] Determine whether there is a mislabeling situation exceeding a third preset range in the first non-video frame image;

[0107] If present, increase the relevance threshold.

[0108] In one example, the third preset range is 50%, that is, more than half of the number of grids with correlation coefficients greater than the correlation threshold are not actual targets.

[0109] In one example, the correlation threshold increases or decreases in a preset step size, and the preset step size is 0.1.

[0110] The above solution solves the problems of huge and tedious workload, high time and labor costs, and low efficiency in manual labeling, and achieves efficient and accurate labeling.

[0111] Based on the same inventive concept, Figure 2 The structure of a device for labeling a target in a picture set provided by an embodiment of the present invention is shown, comprising a judgment module, a first labeling module and a second labeling module, wherein:

[0112] The judging module is used to judge whether the picture set belongs to a video frame picture set or a non-video frame picture set;

[0113] The first labeling module is used to label each first target in the video frame picture set according to the following steps when the result of the judgment module is that the target belongs to the video frame picture set:

[0114] S101, taking a video frame picture in which each first target appears for the first time as a first video frame picture, selecting a first target area in the first video frame picture, taking the first target area as a first target template, and adding label information of the first target area to a first label file of the first video frame picture, wherein the label information includes a position, a size, and a target type number;

[0115] S102, starting from the second video frame picture to the last video frame picture, selecting a search area in the mth (m≥2)th video frame picture, the search area coincides with the center of the first target area of ​​the m-1th video frame picture, and the size of the search area is a preset multiple of the size of the first target area of ​​the m-1th video frame picture, and the preset multiple is greater than 1;

[0116] S103, for different grid sizes, determining in turn whether the size of the search area is divisible by each grid size, if so, dividing the search area into grids according to each grid size, if not, filling the search area so that the size of the filled search area is divisible by each grid size, and then dividing the filled search area into grids according to each grid size;

[0117] S104, scaling all the grids obtained by division to the size of the first target area of ​​the m-1th video frame picture, calculating the correlation coefficient between each scaled grid and the first target template of the m-1th video frame picture, selecting the grid before scaling corresponding to the grid with the largest correlation coefficient as the first target area of ​​the mth video frame picture, using the first target area of ​​the mth video frame picture as the first target template of the mth video frame picture, and adding the label information of the first target area of ​​the mth video frame picture to the first label file of the mth video frame picture;

[0118] S105, drawing a first target annotated area on each video frame picture based on the first label file of each video frame picture, starting from the first video frame picture, determining whether the overlap between the first target annotated area of ​​the nth (n≥1)th video frame picture and the first target actual area is lower than a first preset range, if lower, taking the nth video frame picture as a new first video frame picture, and executing S101-105 for the new first video frame picture and subsequent video frame pictures;

[0119] The second labeling module is used to label each second target in the non-video frame picture set according to the following steps when the result of the judgment module is that the target belongs to the non-video frame picture set:

[0120] S201, taking the non-video frame picture where each second target first appears as the first non-video frame picture, selecting a second target area in the first non-video frame picture, taking the second target area as the second target template, and adding label information of the second target area to the second label file of the first non-video frame picture;

[0121] S202, starting from the second non-video frame picture to the last non-video frame picture, determine whether the size of the kth (k≥2) non-video frame picture is divisible by the size of the second target area, if yes, grid-divide the kth non-video frame picture according to the size of the second target area, if no, fill the kth non-video frame picture so that the size of the filled kth non-video frame picture is divisible by the size of the second target area, and then grid-divide the filled kth non-video frame picture according to the size of the second target area;

[0122] S203, calculating the correlation coefficient between each grid and the second target template, and adding the label information of all grids whose correlation coefficients are greater than the correlation threshold to the second label file of the kth non-video frame image;

[0123] S204. Draw a second target annotation area on each non-video frame picture based on the second label file of each non-video frame picture. Starting from the first non-video frame picture, determine whether the overlap between the second target annotation area of ​​the lth (l≥1) non-video frame picture and the second target actual area is lower than a first preset range. If so, take the lth non-video frame picture as the new first non-video frame picture, and execute S201-204 for the new first non-video frame picture and its subsequent non-video frame pictures.

[0124] Based on the same inventive concept, an embodiment of the present invention provides a computing device, including:

[0125] A memory for storing program instructions;

[0126] The processor is used to call the program instructions stored in the memory and execute the above method according to the obtained program.

[0127] Based on the same inventive concept, an embodiment of the present invention provides a computer-readable non-volatile storage medium, including computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the above method.

[0128] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0132] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0133] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for labeling an object in a picture set, characterized in that: include: Determine whether the picture set belongs to a video frame picture set or a non-video frame picture set; If the picture set belongs to the video frame picture set, each first target in the video frame picture set is labeled according to the following steps: S101, taking a video frame picture in which each first target appears for the first time as a first video frame picture, selecting a first target area in the first video frame picture, taking the first target area as a first target template, and adding label information of the first target area to a first label file of the first video frame picture, wherein the label information includes a position, a size, and a target type number; S102, starting from the second video frame picture to the last video frame picture, selecting a search area in the mth (m≥2)th video frame picture, the search area coincides with the center of the first target area of ​​the m-1th video frame picture, and the size of the search area is a preset multiple of the size of the first target area of ​​the m-1th video frame picture, and the preset multiple is greater than 1; S103, for different grid sizes, determining in turn whether the size of the search area is divisible by each grid size, if so, dividing the search area into grids according to each grid size, if not, filling the search area so that the size of the filled search area is divisible by each grid size, and then dividing the filled search area into grids according to each grid size; S104, scaling all the grids obtained by division to the size of the first target area of ​​the m-1th video frame picture, calculating the correlation coefficient between each scaled grid and the first target template of the m-1th video frame picture, selecting the grid before scaling corresponding to the grid with the largest correlation coefficient as the first target area of ​​the mth video frame picture, using the first target area of ​​the mth video frame picture as the first target template of the mth video frame picture, and adding the label information of the first target area of ​​the mth video frame picture to the first label file of the mth video frame picture; S105, based on the first label file of each video frame picture, draw a first target annotated area on each video frame picture, starting from the first video frame picture, determine whether the overlap between the first target annotated area of ​​the nth (n≥1)th video frame picture and the first target actual area is lower than a first preset range, if lower, take the nth video frame picture as a new first video frame picture, and execute S101-105 for the new first video frame picture and its subsequent video frame pictures; If the picture set belongs to the non-video frame picture set, each second target in the non-video frame picture set is labeled according to the following steps: S201, taking the non-video frame picture in which each second target appears for the first time as the first non-video frame picture, selecting a second target area in the first non-video frame picture, taking the second target area as the second target template, and adding label information of the second target area to the second label file of the first non-video frame picture; S202, starting from the second non-video frame picture to the last non-video frame picture, determining whether the size of the kth (k≥2) non-video frame picture is divisible by the size of the second target area, if so, performing grid division on the kth non-video frame picture according to the size of the second target area, if not, padding the kth non-video frame picture so that the size of the padded kth non-video frame picture is divisible by the size of the second target area, and then performing grid division on the padded kth non-video frame picture according to the size of the second target area; S203, calculating the correlation coefficient between each grid and the second target template, and adding the label information of all grids whose correlation coefficient is greater than the correlation threshold to the second label file of the kth non-video frame picture; S204. Based on the second label file of each non-video frame picture, draw a second target annotation area on each non-video frame picture, and starting from the first non-video frame picture, determine whether the overlap between the second target annotation area of ​​the lth (l≥1) non-video frame picture and the second target actual area is lower than the first preset range; if so, take the lth non-video frame picture as the new first non-video frame picture, and execute S201-204 for the new first non-video frame picture and its subsequent non-video frame pictures.

2. The method according to claim 1, characterized in that: The determining whether the picture set belongs to a video frame picture set or a non-video frame picture set comprises: sorting all the pictures in the picture set based on the picture names; For all the sorted pictures, determine in turn whether there is temporal continuity and content continuity between adjacent pictures; If yes, the picture set belongs to the video frame picture set; If not, the picture set belongs to the non-video frame picture set.

3. The method according to claim 1, characterized in that: The filling of the search area is achieved by filling a black border around the search area.

4. The method according to claim 1, characterized in that: Scaling all the divided grids to the size of the first target area of ​​the m-1th video frame picture is achieved by bilinear interpolation.

5. The method according to claim 1, characterized in that: The calculation formula of the correlation coefficient is as follows: Where r is the correlation coefficient, X i,j is the pixel value at coordinate (i, j) in the grid, is the average pixel value of all coordinates in the grid, Y i,j is the pixel value of the coordinate (i, j) in the target template, is the average pixel value of all coordinates in the target template.

6. The method according to claim 1, characterized in that: Before performing S201-204 on the new first non-video frame picture and its subsequent non-video frame pictures: Determine whether there is a situation in which missing labels exceed a second preset range in the new first non-video frame picture; If so, the correlation threshold is decreased.

7. The method according to claim 1, characterized in that: Before performing S201-204 on the new first non-video frame picture and its subsequent non-video frame pictures: Determine whether there is a mislabeling situation exceeding a third preset range in the first non-video frame picture; If so, the correlation threshold is increased.

8. A device for labeling an image set object, characterized in that: It includes a judgment module, a first marking module and a second marking module, wherein: The judging module is used to judge whether the picture set belongs to a video frame picture set or a non-video frame picture set; The first labeling module is used for labeling each first target in the video frame picture set according to the following steps when the result of the judging module is that the target belongs to the video frame picture set: S101, taking a video frame picture in which each first target appears for the first time as a first video frame picture, selecting a first target area in the first video frame picture, taking the first target area as a first target template, and adding label information of the first target area to a first label file of the first video frame picture, wherein the label information includes a position, a size, and a target type number; S102, starting from the second video frame picture to the last video frame picture, selecting a search area in the mth (m≥2)th video frame picture, the search area coincides with the center of the first target area of ​​the m-1th video frame picture, and the size of the search area is a preset multiple of the size of the first target area of ​​the m-1th video frame picture, and the preset multiple is greater than 1; S103, for different grid sizes, determining in turn whether the size of the search area is divisible by each grid size, if so, dividing the search area into grids according to each grid size, if not, filling the search area so that the size of the filled search area is divisible by each grid size, and then dividing the filled search area into grids according to each grid size; S104, scaling all the grids obtained by division to the size of the first target area of ​​the m-1th video frame picture, calculating the correlation coefficient between each scaled grid and the first target template of the m-1th video frame picture, selecting the grid before scaling corresponding to the grid with the largest correlation coefficient as the first target area of ​​the mth video frame picture, using the first target area of ​​the mth video frame picture as the first target template of the mth video frame picture, and adding the label information of the first target area of ​​the mth video frame picture to the first label file of the mth video frame picture; S105, based on the first label file of each video frame picture, draw a first target annotated area on each video frame picture, starting from the first video frame picture, determine whether the overlap between the first target annotated area of ​​the nth (n≥1)th video frame picture and the first target actual area is lower than a first preset range, if lower, take the nth video frame picture as a new first video frame picture, and execute S101-105 for the new first video frame picture and its subsequent video frame pictures; The second labeling module is used for labeling each second target in the non-video frame picture set according to the following steps when the result of the judging module is that the target belongs to the non-video frame picture set: S201, taking the non-video frame picture in which each second target appears for the first time as the first non-video frame picture, selecting a second target area in the first non-video frame picture, taking the second target area as the second target template, and adding label information of the second target area to the second label file of the first non-video frame picture; S202, starting from the second non-video frame picture to the last non-video frame picture, determining whether the size of the kth (k≥2) non-video frame picture is divisible by the size of the second target area, if so, performing grid division on the kth non-video frame picture according to the size of the second target area, if not, padding the kth non-video frame picture so that the size of the padded kth non-video frame picture is divisible by the size of the second target area, and then performing grid division on the padded kth non-video frame picture according to the size of the second target area; S203, calculating the correlation coefficient between each grid and the second target template, and adding the label information of all grids whose correlation coefficient is greater than the correlation threshold to the second label file of the kth non-video frame picture; S204. Based on the second label file of each non-video frame picture, draw a second target annotation area on each non-video frame picture, and starting from the first non-video frame picture, determine whether the overlap between the second target annotation area of ​​the lth (l≥1) non-video frame picture and the second target actual area is lower than the first preset range; if so, take the lth non-video frame picture as the new first non-video frame picture, and execute S201-204 for the new first non-video frame picture and its subsequent non-video frame pictures.

9. A computing device, characterized in that include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory and execute the method according to any one of claims 1 to 7 according to the obtained program.

10. A computer-readable non-volatile storage medium, characterized in that: The method comprises computer-readable instructions, and when a computer reads and executes the computer-readable instructions, the computer executes the method according to any one of claims 1 to 7.

Images