Neural network data automatic labeling method based on semantic chunk tracking

Abstract

The invention relates to a semantic block tracking-based neural network data automatic labeling method, which comprises the following steps of: aiming at each frame of image imgi in a visual positioning navigation area image data set K1, extracting all semantic targets, obtaining a mask of each semantic target, and obtaining a set M of semantic feature blocks of each frame of image; processing the mask shape of each semantic feature block in the set M to obtain a set M'of each frame of image; according to the set M'of all frame images, feature blocks between any continuous frames are tracked, the data set K1 is traversed based on a traversal strategy, an image annotation data set lambda of a visual positioning navigation area based on a semantic feature block tracking trajectory is obtained, and the set lambda comprises annotation point information of all images in the navigation area; the marking point information of each image comprises center coordinates / angular points and semantic categories of feature blocks in the image. According to the method, the reliability of automatic labeling can be ensured, and rapid and automatic labeling is realized.

Description

Technical Field

[0001] The present invention relates to the field of visual positioning and navigation technology, and in particular to a neural network data automatic labeling method based on semantic block tracking. Background Art

[0002] In the field of visual positioning and navigation applications, pose calculation based on feature point matching is a commonly used SLAM technology, so the extraction of feature points is a very important task. At present, in addition to traditional feature point extraction methods based on SIFT, ORB, SURF, etc., the method of extracting feature points using deep learning networks has also made rapid progress. For example, neural networks such as SuperPoint have been used in SLAM algorithms.

[0003] The training data of deep learning neural networks needs to be annotated in advance. For feature point detection neural networks, the main task of annotation is to annotate the classification label at a certain pixel in the image. The image data containing the pixel position and feature point type is used to train the feature point detection neural network, so that it has the ability to automatically extract feature points in the image and identify the feature point type. If you want to improve the detection rate of the feature point detection neural network, reduce the false detection rate, and improve its generalization performance, you need more accurate annotation and a larger amount of annotation.

[0004] However, labeling is a complex, time-consuming and laborious task. Currently, labeling methods are mainly divided into manual labeling and automatic labeling. The advantages of manual labeling are high accuracy and good consistency, but the disadvantages are low efficiency and high cost. It is applicable to small-scale data sets and tasks with extremely high accuracy requirements. In contrast to manual labeling, automatic labeling technology refers to the process of automatically processing, classifying, identifying and labeling large amounts of data using computer algorithms and models. This technology reduces labor costs, but the disadvantage of automatic labeling is that it relies on algorithm performance and may contain errors or wrong labeling. If the "dirty" training data with the above-mentioned labels is used to train the feature point detection neural network, the detection effect will deteriorate. Therefore, how to reduce the burden of manual labeling and improve the accuracy of the automatic labeling algorithm has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a method for automatic annotation of neural network data based on semantic block tracking, which effectively ensures the reliability of automatic annotation.

[0007] (II) Technical solution

[0008] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0009] In a first aspect, an embodiment of the present invention provides a method for automatic labeling of neural network data based on semantic block tracking, comprising:

[0010] S100, for each frame image img in the visual positioning navigation area image dataset K1 i , extract all semantic targets in each frame image, obtain the mask Mask of each semantic target, and obtain the semantic feature block (x j ,y j ,c j ,Mask j ) of the set M;

[0011] S200, based on the predefined morphological shape, the mask shape of each semantic feature block in the set M of each frame image is processed to obtain the set M' of each frame image, where the set M' includes the feature blocks (x j ,y j ,c j ,Mask j ,G j ), x j ,y j is the center coordinate of the jth feature block, c j is the semantic category of the jth feature block; Mask j is the mask of the jth feature block, G j is the morphological type of the jth feature block;

[0012] S300. According to the set M' of all frame images, the feature blocks between any consecutive frames are tracked, and the data set K1 is traversed based on the traversal strategy to obtain the image annotation data set Λ of the visual positioning navigation area based on the semantic feature block tracking trajectory. The set Λ includes: the annotation point information of each image in the visual positioning navigation area, and the annotation point information of each image includes: the center coordinates / corner points and semantic categories of the feature blocks in the image.

[0013] Optionally, before S100, the method further includes:

[0014] Get the visual positioning navigation area image dataset K1, K1 = {img1, img2, ...img N}, i∈1,2,…,N; N is a natural number greater than 1;

[0015] Alternatively, all paths are walked in the visual positioning navigation area at a speed less than 0.3 m / s to obtain a visual positioning navigation area image dataset K1 of the space above a specified height from the ground.

[0016] Optionally, the S100 includes:

[0017] S101, for each frame image img i , we use the target detection neural network to extract all semantic targets in the image and obtain the set of semantic targets of each image ψ, where ψ is {(x1,y1,c1),(x2,y2,c2),…,(x K ,y K ,c K )}, K is the number of semantic objects in each frame, c is the semantic category, x j ,y j is the pixel coordinate of the center of the jth semantic object in the image;

[0018] S102, input each frame image and the image set Ψ into the SAM neural network to obtain each semantic target x j ,y j Mask j ; Get the semantic feature block of each frame image (x j ,y j ,c j ,Mask j )’s set M.

[0019] Optionally, the S200 includes:

[0020] S201. Based on the morphological shape, the shape of the mask in the set M corresponding to each frame image is filtered to obtain a set M' that only retains the elements in the set M whose masks satisfy a predetermined geometric shape (including but not limited to a circle, an ellipse, a square, a diamond, a star, etc.), thereby obtaining the set M' of each frame image.

[0021] Optionally, the S300 includes:

[0022] If multiple semantic feature blocks detected in multiple consecutive frames of images are the same object in space, the semantic feature blocks of the same object in each frame of image are used as elements to form a set TRACE, and a set R1, Ω consisting of multiple TRACEs is defined. Initially, both the set R1 and the set Ω are empty;

[0023] S301, traverse all images in the K1 set from beginning to end, assuming that the current traversal reaches the kth frame, then for the two consecutive frames img k Image, img k+1 Image, and img k The tth feature block in the image is input into the target tracking neural network to obtain img k+1 Track the pixel area d of the t-th feature block on the image;

[0024] S302, pixel area d and img k+1 The masks of all feature blocks belonging to the image are matched in position;

[0025] S303, if there is a match with the position of the sth feature block, and img k The semantic category c of the tth feature block in the image t with img k+1 The semantic category c of the sth feature block in the image s If the t-th feature block and the s-th feature block are the same, and the difference in size between the mask of the t-th feature block and the mask of the s-th feature block is within a preset range, it is determined that the t-th feature block and the s-th feature block are the same semantic object in two consecutive images, and the tracking is successful; proceed to step S304;

[0026] S304, determine img k Does the tth feature block in the image belong to a TRACE in the set R1? If so, set img k+1 The sth feature block in the image is added to the TRACE. If not, a TRACE is added to the set R1. The new TRACE is initialized with two elements, namely img k The tth feature block in the image and img k+1 The sth feature block in the image;

[0027] S305, check all TRACEs in R1 set, if a TRACE in img k+1 If no semantic feature block is found for successful tracking, it is determined whether the number of elements in TRACE is greater than or equal to Q. If so, the TRACE is moved from the set R1 to the set Ω; Q is a natural number greater than or equal to 3;

[0028] S306 returns to step S301 to traverse the next frame; and

[0029] The TRACEs with the same semantic category in the set Ω obtained after the traversal is completed are merged into the same category; and annotation points are extracted from each semantic feature block of each TRACE in all TRACEs of the merged category, and an image annotation data set Λ as training data in the visual positioning navigation area is generated based on the annotation points and the semantic category after the merged category.

[0030] Optionally, extracting annotation points for each semantic feature block of each TRACE in all TRACEs of the merged category includes: generating annotation points based on different morphological features of the semantic feature block

[0031] If the morphological type of the feature block is a rectangle, the annotation points of the feature block in the image include: the pixel coordinate points of the four right angles of the rectangle;

[0032] If the morphological type of the feature block is circular / elliptical, the annotation points of the feature block in the image include: the pixel coordinate point of the center of the circle;

[0033] The morphological type of the feature block is a triangle / regular shape with angles, and the annotation points of the feature block in the image include: the pixel coordinate point of the center point and the pixel coordinate points of each angle.

[0034] Optionally, the target detection neural network, the SAM neural network, and the target tracking neural network are all open source networks that do not require training;

[0035] The target detection neural network is YOLO / SSD network;

[0036] The target tracking neural network is MDNet.

[0037] In a second aspect, an embodiment of the present invention further provides a neural network data automatic annotation device based on semantic block tracking, comprising:

[0038] The semantic feature block acquisition unit is used to obtain each frame image img in the visual positioning navigation area image dataset K1. i , extract all semantic targets in each frame image, obtain the mask Mask of each semantic target, and obtain the semantic feature block (x j ,y j ,c j ,Mask j ) of the set M;

[0039] The semantic tracking unit is used to process the mask shape of each semantic feature block in the set M of each frame image based on the predefined morphological shape to obtain the set M' of each frame image, where the set M' includes the feature block (x j ,y j ,c j ,Mask j ,G j ), x j ,y j is the center coordinate of the jth feature block, c j is the semantic category of the jth feature block; Mask j is the mask of the jth feature block, G j is the morphological type of the jth feature block;

[0040] According to the set M' of all frame images, the feature blocks between any consecutive frames are tracked, and the data set K1 is traversed based on the traversal strategy to obtain the image annotation data set Λ of the visual positioning navigation area based on the semantic feature block tracking trajectory. The set Λ includes: the annotation point information of each image in the visual positioning navigation area, and the annotation point information of each image includes: the center coordinates and semantic category of the feature block in the image.

[0041] In a third aspect, an embodiment of the present invention further provides a control device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program stored in the memory, specifically executing any of the automatic neural network data labeling methods based on semantic block tracking described in the first aspect above.

[0042] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the automatic labeling method of neural network data based on semantic block tracking as described in any one of the first aspects.

[0043] (III) Beneficial effects

[0044] The method of the present invention is to first segment each frame of the image in the image data set to obtain the segmented semantic target, then obtain the mask of the segmented semantic target, and then use the geometric morphological shape to perform the first verification, and then use the tracking method to perform the second verification to obtain the accurate path information of the data annotation point.

[0045] That is, the method of using continuous multi-frame neural network to track semantic blocks is used to perform secondary verification on the automatically generated semantic block information. Only reliable semantic blocks that have been tracked for a sufficient number of frames will be retained in the training data set, thus ensuring the reliability of automatic labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a method for automatic annotation of neural network data based on semantic block tracking provided by one embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the structure of a neural network data automatic annotation system based on semantic block tracking provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0049] At present, the SAM neural network (Segment Anything Model), with its advanced neural network architecture and extensive training on large data sets, is particularly suitable for image segmentation tasks. It can learn important features in images and be used for tasks such as image classification, target detection, and semantic segmentation. Therefore, based on SAM, a new path has been opened up for automatic image annotation.

[0050] SAM was launched by Meta in 2023 and is a revolutionary image segmentation model. Versatility: SAM is designed to segment anything in an image, from everyday objects to complex scenes, with excellent accuracy and detail. At the same time, in the field of visual positioning and navigation, visual sensors continuously collect images at a high frequency, generally with a frame rate of more than 10Hz. The changes between two adjacent frames of images are not large, and there is a large common view area. Therefore, image tracking methods can be used to use the association of semantic information of multi-frame image data to improve the reliability of the annotation algorithm and reduce "dirty" annotation data.

[0051] Therefore, the object of the present invention is to utilize the characteristics of visual positioning navigation / SLAM images to provide a new neural network data automatic annotation method based on semantic block tracking.

[0052] Embodiment 1

[0053] This embodiment provides a method for automatic labeling of neural network data based on semantic block tracking, that is, a method for automatic labeling of neural network data based on feature point detection of SAM, the execution subject of which can be any control device, and the automatic labeling method includes the following steps:

[0054] Step S100: for each frame image img in the visual positioning navigation area image dataset K1 i , extract all semantic targets in each frame image, obtain the mask Mask of each semantic target, and obtain the semantic feature block (x j ,y j ,c j ,Mask j )’s set M.

[0055] For example, S100 may include the following sub-steps S101 and S102.

[0056] Sub-step S101: for each frame image img i , we use the target detection neural network to extract all semantic targets in the image and get the set of semantic targets of each image, where Ψ is {(x1,y1,c1),(x2,y2,c2),…,(x K ,y K ,c K )}, K is the number of semantic objects in each frame, c is the semantic category, x j ,y j is the pixel coordinate of the center of the jth semantic object in the image;

[0057] Sub-step S102: Input each frame image and the image set ∑ into the SAM neural network to obtain each semantic target x j ,yj Mask j ; Get the semantic feature block of each frame image (x j ,y j ,c j ,Mask j )’s set M.

[0058] Step S200: Process the mask shape of each semantic feature block in the set M of each frame image based on the predefined morphological shape to obtain a set M' of each frame image, wherein the set M' includes the feature blocks (x j ,y j ,c j ,Mask j ,G j ), x j ,y j is the center coordinate of the jth feature block, c j is the semantic category of the jth feature block; Mask j is the mask of the jth feature block, G j is the morphological type of the jth feature block.

[0059] For example, step S200 of this embodiment may include the following sub-step S201.

[0060] S201. Based on the morphological shape, the shape of the mask in the set M corresponding to each frame image is filtered to obtain a set M' that only retains the elements in the set M whose masks satisfy a predetermined geometric shape (including but not limited to a circle, an ellipse, a square, a diamond, a star, etc.), thereby obtaining the set M' of each frame image.

[0061] Step S300, according to the set M' of all frame images, track the feature blocks between any consecutive frames, traverse the data set K1 based on the traversal strategy, and obtain the image annotation data set Λ of the visual positioning navigation area based on the semantic feature block tracking trajectory, the set Λ includes: the annotation point information of each image in the visual positioning navigation area, the annotation point information of each image includes: the center coordinates of the feature block in the image (or the center coordinates and corner points, or the corner points of the feature block) and the semantic category.

[0062] In this embodiment, the annotation points of each image in the set Λ are the feature points of the mask in the image. Usually, the coordinates and semantic categories of targets (i.e., the same object) appearing in more than Q frames in Q frames are used as automatic annotation points of the training data, and Q is a natural number greater than or equal to 3.

[0063] For example, step S300 may include the following sub-steps:

[0064] If multiple semantic feature blocks detected in multiple consecutive frames of images are the same object in space, the semantic feature blocks of the same object in each frame of image are used as elements to form a set TRACE, and a set R1, Ω consisting of multiple TRACEs is defined. Initially, both the set R1 and the set Ω are empty;

[0065] S301, traverse all images in the K1 set from beginning to end, assuming that the current traversal reaches the kth frame, then for the two consecutive frames img k Image, img k+1 Image, and img k The tth feature block in the image is input into the target tracking neural network to obtain img k+1 Track the pixel area d of the t-th feature block on the image;

[0066] S302, pixel area d and img k+1 The masks of all feature blocks belonging to the image are matched in position;

[0067] S303, if there is a match with the position of the sth feature block, and img k The semantic category c of the tth feature block in the image t with img k+1 The semantic category c of the sth feature block in the image s If the t-th feature block and the s-th feature block are the same, and the difference in size between the mask of the t-th feature block and the mask of the s-th feature block is within a preset range, it is determined that the t-th feature block and the s-th feature block are the same semantic object in two consecutive images, and the tracking is successful; proceed to step S304;

[0068] S304, determine img k Does the tth feature block in the image belong to a TRACE in the set R1? If so, set img k+1 The sth feature block in the image is added to the TRACE. If not, a TRACE is added to the set R1. The new TRACE is initialized with two elements, namely img k The tth feature block in the image and img k+1 The sth feature block in the image;

[0069] S305, check all TRACEs in R1 set, if a TRACE in img k+1 If no semantic feature block is found for successful tracking, it is determined whether the number of elements in TRACE is greater than or equal to Q. If so, the TRACE is moved from the set R1 to the set Ω; Q is a natural number greater than or equal to 3;

[0070] S306 returns to step S301 to traverse the next frame; and

[0071] The TRACEs with the same semantic category in the set Ω obtained after the traversal is completed are merged into the same category; and annotation points are extracted from each semantic feature block of each TRACE in all TRACEs of the merged category, and an image annotation data set Λ as training data in the visual positioning navigation area is generated based on the annotation points and the semantic category after the merged category.

[0072] In this embodiment, extracting annotation points for each semantic feature block of each TRACE in all TRACEs of the merged category includes: generating annotation points based on different morphological features of the semantic feature block;

[0073] If the morphological type of the feature block is a rectangle, the annotation points of the feature block in the image include: the pixel coordinate points of the four right angles of the rectangle;

[0074] If the morphological type of the feature block is circular / elliptical, the annotation points of the feature block in the image include: the pixel coordinate point of the center of the circle;

[0075] The morphological type of the feature block is a triangle / regular shape with angles, and the annotation points of the feature block in the image include: the pixel coordinate point of the center point and the pixel coordinate points of each angle.

[0076] The method of this embodiment uses a morphological method to screen the automatically generated semantic blocks, and only retains semantic feature blocks with regular shapes and similar sizes for successful tracking and matching. During the processing, the automatically generated semantic feature block information is secondary verified, and only reliable semantic blocks that have been tracked for a sufficient number of frames will be retained in the training data set, thereby ensuring the reliability of automatic labeling.

[0077] Embodiment 2

[0078] In addition, this embodiment provides a method for automatic labeling of neural network data based on feature point detection of SAM, the execution subject of which can be any control device such as the device where the scheduling system of the robot is located, and the automatic labeling method includes the following steps:

[0079] S1. Data collection is to obtain the visual positioning navigation area image dataset K1.

[0080] For example, in this embodiment, the camera can be moved by hand, or the camera can be installed on a mobile robot to operate the mobile robot to move, or the camera can be installed on a mobile backpack and other methods.

[0081] The camera in this embodiment walks once in the visual positioning navigation area at a relatively slow speed (such as 0.3 m / s), and the area recorded by the camera needs to cover the walking path of the mobile robot. That is, walk all paths in the visual positioning navigation area at a speed less than 0.3 m / s to obtain the visual positioning navigation area image dataset K1 above the specified height from the ground.

[0082] It is understandable that in order to better capture images, the camera shooting angle in this embodiment can be toward the upper and middle space (such as the camera shooting angle is tilted upward by 20°-80° relative to the horizontal direction) to avoid occlusion caused by dynamic objects and environment in the near-ground area.

[0083] Without loss of generality, assume that a total of N images are collected and define the set as K1, K1 = {img1, img2, …img N}, i∈1,2,…,N; N is a natural number greater than 1.

[0084] S2. Extract the semantic feature blocks of each frame image.

[0085] For all images in the dataset K1, the semantic feature blocks of each image are extracted.

[0086] This embodiment shows a step of extracting semantic feature blocks in an image, which is specifically as follows:

[0087] S201. For any frame image img i , i∈1,2,…,N, the image is input into the target detection neural network (such as YOLO, SSD network), the target detection neural network outputs the segmented semantic targets of each form, obtains the pixel coordinates of the semantic target of each form (such as the pixel coordinates of the center of the morphological area), and obtains the set of semantic targets extracted for each image Ψ, Ψ is {(x1,y1,c1),(x2,y2,c2),…,(x K ,y K ,c K )}.

[0088] Without loss of generality, define img i The number of semantic objects extracted is K, and the set Ψ is {(x1,y1,c1),(x2,y2,c2),…,(x K ,y K ,c K )}, where x, y are the pixel coordinates of the semantic target in the image, c represents the category of the semantic target, i.e., the semantic category; K is the number of semantic targets in each frame of the image.

[0089] S202. For each element (x j ,y j ,c j ), j∈1,2,…,K.

[0090] The pixel coordinates (x j ,y j) is used as the input of the SAM neural network, and a corresponding mask Mask is calculated and output through the SAM neural network. j , the mask is the pixel point (x j ,y j ) semantic segmentation result. Without loss of generality, define each (x j ,y j ,c j ,Mask j ) is a semantic feature block.

[0091] In practical applications, the image img i and image img i The set Ψ is input into the SAM neural network together, and the SAM neural network outputs the image img i The set M.

[0092] It is understandable that the target detection neural network can detect a certain light in the image, but it can only detect a certain pixel point. By inputting the pixel point value into the SAM network, all the pixels of the entire light can be segmented out in the image to generate a light mask.

[0093] S3. Based on the predefined morphological shape, the mask shape of each semantic feature block in the set M of each frame image is processed to obtain the set M' of each frame image.

[0094] That is, for all semantic feature blocks in all the above images, the results are preprocessed by morphology according to the shapes of their masks.

[0095] Based on the morphological shape, the shape of the mask in the set M corresponding to each frame image is filtered to obtain a set M' that only retains the elements in the set M whose masks satisfy a predetermined geometric shape (including but not limited to a circle, an ellipse, a square, a diamond, a star, etc.), thereby obtaining a set M' for each frame image.

[0096] In this embodiment, the geometric shape of the object in the visual positioning navigation area can be pre-identified to obtain the geometric morphological shape of the object; then based on the morphological shape, the shape of the mask in the set M corresponding to each frame image is matched and filtered to obtain the set M' of each frame image.

[0097] For example, the morphological shape may include: circle, ellipse, rectangle, parallelogram, five-pointed star, column, etc.

[0098] The set M' includes the feature blocks (x j ,y j ,c j ,Mask j ,JH j ), x j ,yj is the center coordinate of the jth feature block, c j is the semantic category of the jth feature block; Mask j is the mask of the jth feature block, JH j is the morphological type of the jth feature block.

[0099] In this embodiment, the mask may be retained as semantic feature blocks of several pre-given regular geometric shapes, such as several common categories such as rectangle and ellipse, and complex and irregularly shaped semantic feature blocks may be screened out.

[0100] That is to say, in this embodiment, the tracking of the visual positioning navigation area is first verified by step S3, and the second level verification is performed in step S4 to improve the reliability of verification and tracking matching.

[0101] S4. Perform tracking, matching and detection on the semantic feature blocks between any two consecutive frames.

[0102] Without loss of generality, assume that the current frame is img k , then the next frame image is img k+1 , for img k Any feature block t, (x t ,y t ,c t ,Mask t ),

[0103] Img k Image, img k+1 Image, feature block t, input target tracking neural network (such as SiameseNetwork, MDNet), target tracking neural network output in img k+1 The pixel area d tracked on the image,

[0104] If the pixel area d happens to coincide with img k+1 The position of the mask of a certain feature block s detected basically coincides, and c t =c s , and the mask shapes and sizes of feature block t and feature block s are similar, then feature block t and feature block s are considered to be the same semantic object in the two consecutive images, and the semantic feature block tracking is successful. As mentioned above, all successfully matched semantic feature blocks between any two consecutive frames can be found.

[0105] S5. Traverse and generate semantic feature blocks TRACE.

[0106] If multiple semantic feature blocks detected in multiple consecutive frames of images are the same object in space, the semantic feature blocks of the same object in each frame of the image are used as elements to form a set TRACE, and a set R1, Ω consisting of multiple TRACEs is defined. Initially, both the set R1 and the set Ω are empty.

[0107] That is to say, the semantic feature block set R1 currently being tracked is defined as the feature block set that is successfully tracked as Ω, and the elements in these two feature block sets are called TRACEs. Each TRACE corresponds to the same object seen in multiple frames of images. For example, if the third semantic feature block in the first frame, the fifth feature block in the second frame, and the fourth feature block in the third frame are the same object, they can constitute a TRACE.

[0108] Before the traversal begins, R1 and Ω are both empty (i.e., there is no TRACE). All images are traversed backwards from the second frame in sequence. Without loss of generality, it is assumed that the current traversal reaches the nth frame.

[0109] S.501 traverses all semantic feature blocks of the current nth frame. If the mth semantic feature block of the nth frame image is successfully matched with the rth semantic feature block of the n-1th frame, it corresponds to the content in the above S4.

[0110] Then check whether the rth semantic feature block of the n-1th frame is located in a TRACE in the set R1. If so, add the mth semantic feature block of the nth frame image to the TRACE.

[0111] S.502 traverses all TRACEs in set R1. If a TRACE does not find a matching semantic feature block in the current frame, it determines whether the number of elements in the TRACE is greater than the first threshold. If so, the TRACE is moved to the semantic feature block set Ω where tracking is successful. Otherwise, the TRACE is directly deleted.

[0112] S6. For all semantic feature blocks in all TRACEs in the semantic feature block set Ω, extract the annotation points corresponding to the semantic feature blocks,

[0113] Among them, when the mask is a rectangle, the semantic feature block of the mask can extract four right angles as annotation points. For the semantic feature blocks with no corner points such as circles and ellipses, the centers of the circles are extracted as annotation points.

[0114] S7. Merge the TRACE data of the same category in the set Ω into data of the same category, wherein the annotation points extracted according to step S6 are the annotation points of the training data, thereby generating an image annotation data set Λ as training data in the visual positioning navigation area based on the annotation points and the semantic categories after the merged categories.

[0115] In this embodiment, the first verification is to use morphological methods to screen the automatically generated semantic blocks, retaining only semantic blocks with regular shapes and similar sizes, and then use the tracking network to perform tracking and matching to achieve the second verification, effectively ensuring the reliability of automatic labeling.

[0116] Embodiment 3

[0117] This embodiment also provides a neural network data automatic annotation device based on semantic block tracking, the automatic annotation device includes: a semantic feature block acquisition unit and a semantic tracking unit;

[0118] The semantic feature block acquisition unit is used to locate each frame image img in the visual positioning navigation area image dataset K1. i , extract all semantic targets in each frame image, obtain the mask Mask of each semantic target, and obtain the semantic feature block (x j ,y j ,c j ,Mask j ) of the set M;

[0119] The semantic tracking unit is used to process the mask shape of each semantic feature block in the set M of each frame image based on the predefined morphological shape to obtain the set M' of each frame image, the set M' includes the feature block (x j ,y j ,c j ,Mask j ,G j ), x j ,y j is the center coordinate of the jth feature block, c j is the semantic category of the jth feature block; Mask j is the mask of the jth feature block, G j is the morphological type of the jth feature block;

[0120] According to the set M' of all frame images, the feature blocks between any consecutive frames are tracked, and the data set K1 is traversed based on the traversal strategy to obtain the image annotation data set Λ of the visual positioning navigation area based on the semantic feature block tracking trajectory. The set Λ includes: the annotation point information of each image in the visual positioning navigation area, and the annotation point information of each image includes: the center coordinates and semantic category of the feature block in the image.

[0121] In this embodiment, the visual navigation area applied to the robot realizes the tracking of data annotation points, which mainly realizes automatic data annotation points through two verifications. For example, the first verification adopts the morphological method to screen the automatically generated semantic blocks, and only retains the semantic blocks with regular shapes and similar sizes. Then, the tracking network is used to perform tracking and matching to realize the second verification, which effectively ensures the reliability of automatic annotation.

[0122] In addition, an embodiment of the present invention also provides a control device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program stored in the memory, specifically performing the steps of the method for automatic labeling of neural network data based on semantic block tracking described in any of the above embodiments.

[0123] On the other hand, an embodiment of the present invention further provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the automatic labeling method of neural network data based on semantic block tracking as described in any of the above embodiments are implemented.

[0124] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0126] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0127] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0128] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0129] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. A method for automatic annotation of neural network data based on semantic block tracking, characterized in that: include: S100, for each frame image img in the visual positioning navigation area image dataset K1 i , extract all semantic targets in each frame image, obtain the mask Mask of each semantic target, and obtain the semantic feature block (x j ,y j , c j , Mask j ) of the set M; S200, based on the predefined morphological shape, the mask shape of each semantic feature block in the set M of each frame image is processed to obtain the set M' of each frame image, where the set M' includes the feature blocks (x j ,y j , c j , Mask j , G j ), x j ,y j is the center coordinate of the jth feature block, c j is the semantic category of the jth feature block; Mask j is the mask of the jth feature block, G j is the morphological type of the jth feature block; S300. According to the set M' of all frame images, the feature blocks between any consecutive frames are tracked, and the data set K1 is traversed based on the traversal strategy to obtain the image annotation data set Λ of the visual positioning navigation area based on the semantic feature block tracking trajectory. The set Λ includes: the annotation point information of each image in the visual positioning navigation area, and the annotation point information of each image includes: the center coordinates and semantic category of the feature block in the image.

2. The method according to claim 1, characterized in that Before S100, the method further includes: Get the visual positioning navigation area image dataset K1, K1 = {img1, img2, ...img N }, i∈1,2,...,N; N is a natural number greater than 1; Alternatively, all paths are walked in the visual positioning navigation area at a speed less than 0.3 m / s to obtain a visual positioning navigation area image dataset K1 of the space above a specified height from the ground.

3. The method according to claim 1, characterized in that The S100 includes: S10 1. For each frame image img i , we use the target detection neural network to extract all semantic targets in the image and get the set of semantic targets of each image, where Ψ is {(x1, y1, c1), (x2, y2, c2), ..., (x K ,y K , c K ), K is the number of semantic objects in each frame, c is the semantic category, x j ,y j is the pixel coordinate of the center of the jth semantic object in the image; S102, input each frame image and the image set Ψ into the SAM neural network to obtain each semantic target x j ,y j Mask j ; Get the semantic feature block of each frame image (x j ,y j , c j , Mask j )’s set M.

4. The method according to claim 1, characterized in that: The S200 includes: S201 , based on morphological shapes, filtering the shapes of masks in the set M corresponding to each frame of image, obtaining a set M' that only retains elements of the set M whose masks satisfy a predetermined geometric shape, and obtaining a set M' for each frame of image.

5. The method according to claim 1, characterized in that The S300 includes: If multiple semantic feature blocks detected in multiple consecutive frames of images are the same object in space, the semantic feature blocks of the same object in each frame of image are used as elements to form a set TRACE, and a set R1, Ω consisting of multiple TRACEs is defined. Initially, both the set R1 and the set Ω are empty; S301, traverse all images in the K1 set from beginning to end, assuming that the current traversal reaches the kth frame, then for the two consecutive frames img k Image, img k+1 Image, and img k The tth feature block in the image is input into the target tracking neural network to obtain img k+1 Track the pixel area d of the t-th feature block on the image; S302, pixel area d and img k+1 The masks of all feature blocks belonging to the image are matched in position; S303, if there is a match with the position of the sth feature block, and img k The semantic category c of the tth feature block in the image t with img k+1 The semantic category c of the sth feature block in the image s If the t-th feature block and the s-th feature block are the same, and the difference in size between the mask of the t-th feature block and the mask of the s-th feature block is within a preset range, it is determined that the t-th feature block and the s-th feature block are the same semantic object in two consecutive images, and the tracking is successful; proceed to step S304; S304, determine img k Does the tth feature block in the image belong to a TRACE in the set R1? If so, set img k+1 The sth feature block in the image is added to the TRACE. If not, a TRACE is added to the set R1. The new TRACE is initialized with two elements, namely img k The tth feature block in the image and img k+1 The sth feature block in the image; S305, check all TRACEs in R1 set, if a TRACE in img k+1 If no semantic feature block is found for successful tracking, it is determined whether the number of elements in TRACE is greater than or equal to Q. If so, the TRACE is moved from the set R1 to the set Ω; Q is a natural number greater than or equal to 3; S306 returns to step S301 to traverse the next frame; and The TRACEs with the same semantic category in the set Ω obtained after the traversal is completed are merged into the same category; and annotation points are extracted from each semantic feature block of each TRACE in all TRACEs of the merged category, and an image annotation data set Λ as training data in the visual positioning navigation area is generated based on the annotation points and the semantic category after the merged category.

6. The method according to claim 5, extracting annotation points for each semantic feature block of each TRACE in all TRACEs of the merged category, comprising: Generate annotation points based on different morphological features of semantic feature blocks. If the morphological type of the feature block is a rectangle, the annotation points of the feature block in the image include: the pixel coordinate points of the four right angles of the rectangle; If the morphological type of the feature block is circular / elliptical, the annotation points of the feature block in the image include: the pixel coordinate point of the center of the circle; The morphological type of the feature block is a triangle / regular shape with angles, and the annotation points of the feature block in the image include: the pixel coordinate point of the center point and the pixel coordinate points of each angle.

7. The method according to claim 1, characterized in that The target detection neural network, SAM neural network, and target tracking neural network are all open source networks that do not require training; The target detection neural network is YOLO / SSD network; The target tracking neural network is MDNet.

8. A neural network data automatic annotation device based on semantic block tracking, characterized in that: include: The semantic feature block acquisition unit is used to obtain each frame image img in the visual positioning navigation area image dataset K1. i , extract all semantic targets in each frame image, obtain the mask Mask of each semantic target, and obtain the semantic feature block (x j ,y j , c j , Mask j ) of the set M; The semantic tracking unit is used to process the mask shape of each semantic feature block in the set M of each frame image based on the predefined morphological shape to obtain the set M' of each frame image, where the set M' includes the feature block (x j ,y j , c j , Mask j , G j ), x j ,y j is the center coordinate of the jth feature block, c j is the semantic category of the jth feature block; Mask j is the mask of the jth feature block, G j is the morphological type of the jth feature block; According to the set M' of all frame images, the feature blocks between any consecutive frames are tracked, and the data set K1 is traversed based on the traversal strategy to obtain the image annotation data set Λ of the visual positioning navigation area based on the semantic feature block tracking trajectory. The set Λ includes: the annotation point information of each image in the visual positioning navigation area, and the annotation point information of each image includes: the center coordinates and semantic category of the feature block in the image.

9. A control device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program stored in the memory, specifically executing the automatic labeling method for neural network data based on semantic block tracking as described in any one of claims 1 to 7.

10. A computer-readable storage medium, comprising: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for automatic labeling of neural network data based on semantic block tracking as described in any one of claims 1 to 7 are implemented.

Images