Marking recognition method and device

Abstract

The present application discloses a marker recognition method and device, which relates to the field of image processing technology. The method includes: obtaining a marker image containing a marker in a real environment; obtaining the corner points of the target number in the marker image; determining the area where the marker is located from the marker image based on the corner points of the target number; extracting multiple image feature points in the image corresponding to the area where the marker is located based on a pre-trained key point detection model; obtaining the types of multiple image feature points, and identifying the marker based on the multiple image feature points and the types of image feature points to obtain a recognition result, which is used to track and locate the marker. In this way, the area where the marker is located can be determined more accurately, and more image feature points can be obtained, and the marker can be identified in combination with the type of image feature points, so that a higher posture accuracy of the marker can be obtained, and the robustness of marker recognition can also be greatly improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a marker recognition method and device. Background Art

[0002] Fiducial markers are artificial visual features designed for automatic detection. In related technologies, in order to achieve object recognition, positioning and tracking, corresponding fiducial markers are usually set on the corresponding objects, and the identity and position information of the objects are obtained through identification and positioning of the corresponding fiducial markers.

[0003] In the related art, the key points of the reference markers are generally extracted based on edge detection and image binarization, and then the identity and posture information of the reference markers are identified based on the extracted key points. However, the posture information identified in this way has the problem of poor accuracy. Summary of the invention

[0004] In view of this, the present application proposes a marking recognition method and device.

[0005] In a first aspect, an embodiment of the present application provides a marker recognition method, the method comprising: obtaining a marker image containing a marker in a real environment; obtaining a target number of corner points in the marker image; determining an area where the marker is located from the marker image based on the target number of corner points; extracting multiple image feature points in an image corresponding to the area where the marker is located based on a pre-trained key point detection model; obtaining types of the multiple image feature points, the number of which is determined based on the marker type of the marker; identifying the marker based on the multiple image feature points and the types of the image feature points to obtain a recognition result, and the recognition result is used to track and locate the marker.

[0006] In the second aspect, the embodiment of the present application provides a marker recognition device, the device comprising: an image acquisition module, a corner point acquisition module, a region determination module, a feature point extraction module, a type acquisition module and a marker recognition module. The image acquisition module is used to acquire a marker image containing a marker in a real environment; the corner point acquisition module is used to acquire a target number of corner points in the marker image; the region determination module is used to determine the region where the marker is located from the marker image based on the target number of corner points; the feature point extraction module is used to extract multiple image feature points in the image corresponding to the region where the marker is located based on a pre-trained key point detection model; the type acquisition module is used to acquire the types of the multiple image feature points, the number of which is determined based on the marker type of the marker; the marker recognition module is used to identify the marker based on the multiple image feature points and the types of the image feature points to obtain a recognition result, and the recognition result is used to track and locate the marker.

[0007] In a third aspect, an embodiment of the present application provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the tag recognition method provided in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the tag recognition method provided in the first aspect.

[0009] In the solution provided by the present application, a marked image containing marks in a real environment is obtained; corner points of a target number in the marked image are obtained; based on the corner points of the target number, the area where the mark is located is determined from the marked image; based on a pre-trained key point detection model, multiple image feature points in the image corresponding to the area where the mark is located are extracted; the types of multiple image feature points are obtained, and the number of types is determined based on the mark type of the mark; based on multiple image feature points and the types of image feature points, the mark is identified to obtain a recognition result, and the recognition result is used to track and locate the mark. In this way, by obtaining the corner points in the marked image to determine the area where the mark is located, it is possible to more accurately and quickly determine the area where the mark is located; based on this, more and denser image feature points can be extracted to achieve higher posture accuracy; and in combination with the types of image feature points, the mark is identified, which also greatly improves the robustness of mark recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A flow chart of a marking recognition method provided in an embodiment of the present application is shown.

[0012] Figure 2 A schematic diagram of a reference mark provided in an embodiment of the present application is shown.

[0013] Figure 3 A schematic diagram of the identification process provided in an embodiment of the present application is shown.

[0014] Figure 4 Shows Figure 2 The flowchart of sub-steps of step S260 in one implementation is shown.

[0015] Figure 5 A flow chart of a marking recognition method provided in yet another embodiment of the present application is shown.

[0016] Figure 6 A schematic diagram showing key feature points of a mark provided by an embodiment of the present application is shown.

[0017] Figure 7 A schematic diagram showing key feature points of a mark provided by another embodiment of the present application is shown.

[0018] Figure 8 A schematic flow chart of a marking recognition method provided in yet another embodiment of the present application is shown.

[0019] Fig. 9 A schematic diagram of a partial pattern of a mark provided in an embodiment of the present application is shown.

[0020] Fig.10 A flow chart of a marking recognition method provided in yet another embodiment of the present application is shown.

[0021] Fig.11 A schematic diagram of image feature points marked by RuneTag provided in an embodiment of the present application is shown.

[0022] Fig.12 A schematic diagram of image feature points marked by RuneTag provided in another embodiment of the present application is shown.

[0023] Fig.13 A schematic diagram of image feature points marked by TopoTag provided in an embodiment of the present application is shown.

[0024] Fig.14 A schematic diagram of image feature points marked by a TopoTag provided in another embodiment of the present application is shown.

[0025] Fig.15 A schematic diagram showing image feature points marked by an AprilTag provided in an embodiment of the present application is shown.

[0026] Fig.16 A schematic diagram showing image feature points marked by an AprilTag provided in another embodiment of the present application is shown.

[0027] Fig.17 It is a block diagram of a mark recognition device provided according to another embodiment of the present application.

[0028] Fig.18 It is a block diagram of a computer device for executing a tag recognition method according to an embodiment of the present application.

[0029] Fig.19 It is a storage unit of an embodiment of the present application for storing or carrying a program code for implementing a marking recognition method according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0031] In the related art, the key points of the reference markers are generally extracted based on edge detection and image binarization, and then the identity and posture information of the reference markers are identified based on the extracted key points. However, the posture information identified by this method has a problem of poor accuracy.

[0032] In view of the above problems, the inventor proposes a marker recognition method and device, which can determine the area where the marker is located in the marker image based on the corner points in the marker image by a computer device; extract multiple image feature points, and recognize the marker based on the types of the multiple image feature points to obtain a marker recognition result. The content is described in detail below.

[0033] Please refer to Figure 1 , Figure 1 A flowchart of a tag recognition method provided in an embodiment of the present application is shown below. Figure 1 The tag recognition method provided in the embodiment of the present application is described in detail. The tag recognition method may include the following steps:

[0034] Step S210: Acquire a marker image including markers in the real environment.

[0035] In this embodiment, the mark is a pre-set reference mark. In applications such as augmented reality, user gesture input of handheld devices, and robot navigation, the mark can be printed and placed in the scene of the real environment. Then, by obtaining a mark image containing the mark in the real environment, the mark in the mark image can be identified and its position and posture can be recognized, thereby tracking and positioning the mark. Compared with natural objects without mark designs, reference marks can provide more and more reliable feature information in computer vision, thereby improving the accuracy of tracking and positioning. The way to obtain the mark image can be that the computer device collects it through the image acquisition device configured by itself; or it can establish a communication connection with an external image acquisition device, and the external image acquisition device collects images of the real environment and sends the collected images to the computer device. This embodiment does not limit this.

[0036] Among them, the benchmark marks include but are not limited to Figure 2 The five tags shown in the figure are ARToolkitPlus, Augmented RealityUniversity of Cordoba (Aruco)Tag, AprilTag, TopoTag and RuneTag. In the prior art, the tags are generally detected by traditional image processing techniques such as edge detection, spot detection or image binarization. However, the aforementioned traditional image processing techniques can only support relatively simple reference tags. Figure 2 The ARToolkitPlus, Aruco, and AprilTag in the ARToolkitPlus use black and white chessboard appearance. They detect their quadrilateral boundaries through edge detection and analysis of straight lines, decode through binarization of small areas, and estimate the pose information of the marker using the four corners of the marker. The accuracy is poor, especially when the marker image containing the marker is noisy or motion blurred, which leads to inaccurate recognition results of the marker. Figure 2 For TopoTag in the figure, each area in the chessboard layout represents a key point and a bit (such as coding bit 1 or coding bit 0). However, due to the simplicity of the appearance of this type of tag, the tag needs to have rotational asymmetry to ensure the uniqueness of the tag and avoid posture ambiguity. This leads to the limitations of the tag and makes the design of the encoding system more complicated; and Figure 2The key points of tags such as RuneTag in the image are irregularly distributed, which causes the bits corresponding to each key point to be misaligned and decoded, which also reduces the accuracy of the tag recognition results. It can be seen that the appearance design of tags in the prior art is generally black and white and has a relatively simple shape, which increases the possibility of confusion between the tag and the environmental elements, thereby increasing the difficulty of tag recognition and reducing the accuracy of the tag recognition results.

[0037] Based on this, in this embodiment, a DeepTag general framework for reference markers is provided. The DeepTag general framework can be used to detect reference markers with simple appearance in the above-mentioned prior art, and define denser key points, so as to obtain higher posture accuracy for the markers. The robustness of marker detection is also greatly improved by using sub-pixel information. At the same time, more complex local patterns are provided to design new and more complex reference markers to reduce the possibility of confusion between markers and environmental elements in the real environment, reduce the difficulty of marker recognition, and improve the accuracy of marker recognition results. In addition, the DeepTag general framework detects markers in a bottom-up manner, based on the Single Shot MultiboxDetector (SSD) algorithm, using a single convolutional network to predict the bounding box of the marker and its class probability, and regressing key points and digital symbols from local shapes. Traditional low-order image processing technology is not used to detect edges or spots, and the bounding box assumptions of resampling pixels or features are not used, so as to achieve faster calculation speed and improve the efficiency and accuracy of marker recognition.

[0038] Step S220: Obtaining a target number of corner points in the marked image.

[0039] In this embodiment, the corner point is an extreme point, which can be understood as a point with particularly prominent attributes in a certain aspect. For an image, a corner point can be understood as a point in the image where the gradient value and gradient direction have a very high rate of change, or can be understood as the intersection of two or more edges of the image, that is, the connection point of the object's contour line. Based on this, after obtaining a marked image containing a mark in a real environment, the area where the mark is located can be determined by extracting the corner points of the target number in the marked image, that is, by extracting the connection points of the marked contour line, the area where the mark is located can be determined. Among them, the target number can be pre-set or adjusted according to different application scenarios, and this embodiment does not limit this; the corner points in the extracted marked image can be extracted by a variety of detection algorithms, and the detection algorithms include but are not limited to corner point detection based on grayscale images, corner point detection based on binary images, corner point detection based on contour curves, etc.

[0040] Since corner points can effectively reduce the amount of information while retaining important features of image graphics, the information content is very high, which effectively improves the speed of calculation; and even if the viewing angle of the marked image changes, the accuracy of the marked area determined based on the corner points can still be guaranteed due to the stable nature of the corner points.

[0041] Step S230: Based on the target number of corner points, determine the area where the mark is located from the mark image.

[0042] In this embodiment, since the corner point detection algorithm may have certain errors, the target number of corner points obtained may not be completely the corner points on the contour curve of the marker, and may also include corner points on the contour curves of other objects in the real environment. Therefore, the area where the marker is located can be further determined from the marker image based on the target number of corner points.

[0043] Specifically, obtain the correlation between each corner point of the target number of corner points and the bounding box of the preset area, and obtain the cosine value of the angle between the direction vector of each corner point pointing to the center of the mark and the preset direction vector; determine whether the correlation between each corner point and the bounding box value of the preset area is less than the preset correlation, and whether the cosine value of the angle between the direction vector of each corner point pointing to the center of the mark and the preset direction vector is greater than a preset value; if not, obtain the corner points that do not meet the above judgment conditions as unqualified corner points, and screen out the unqualified corner points from the target number of corner points to obtain multiple qualified corner points; based on the multiple qualified corner points, determine the area where the mark is located.

[0044] Specifically, the corner points in the marked image are obtained, and the area where the mark is located is determined based on the corner points, that is, Figure 3 In the stage 1 shown, a region of interest (ROI) is detected on the marked image. The ROI detection model can be pre-trained. In practical applications, the pre-trained ROI detection model can be directly called to detect the marked area in the marked image, that is, to detect the marked ROI.

[0045] In this embodiment, the ROI detection model is trained based on the single-stage multi-box predictor (Single Shot Detector, SSD) and the single-scale predictor. Among them, the core of SSD is to use a small convolution filter applied to the feature map to predict the category scores and box offsets of a fixed set of default bounding boxes, generate a fixed-size bounding box set and the score of object class instances in these boxes, and the score can be used as the confidence. First, the classifier can be trained based on a large number of training samples until the preset conditions are met, wherein the preset conditions can be: the total loss value is less than the preset value, the total loss value no longer changes, or the number of training times reaches the preset number of times, etc.; the training samples include positive samples and negative samples, the positive samples are images including various labels, and the negative samples are images without labels. It can be understood that after the classifier is iteratively trained for multiple training cycles based on a large number of training samples, each training cycle includes multiple iterative trainings, and the parameters are continuously optimized so that the above total loss value becomes smaller and smaller, and finally becomes a fixed value, or is less than the above preset value. At this time, it means that the initial model has converged; of course, it can also be determined that the initial model has converged after the number of training times reaches the preset number of times. At this time, the classifier in the initial model can be used as an ROI detection model for detecting the marked ROI. Among them, the preset value and the preset number of times are pre-set, and their values ​​can also be adjusted according to different application scenarios. This embodiment does not limit this.

[0046] Among them, the total loss function in the ROI detection stage is the sum of the loss of the SSD predictor, the loss of the single-scale predictor, and the loss of the mask. The total loss function L ROI It can be calculated by the following formula:

[0047] L ROI =L box +L corner +L mask

[0048] Among them, L box represents the loss of the SSD predictor, that is, the loss when predicting the boundary of the label, L corner represents the loss of the single-scale predictor, that is, the loss of predicting corner points, L mask Represents mask loss.

[0049] L box It can be calculated by the following formula:

[0050]

[0051] N is the total number of matched borders, L conf (c (b)) is the softmax loss value of the predicted border. The softmax loss value can be calculated by the following formula:

[0052]

[0053] Represents the matching of the i-th default box / corner point to the j-th real default box / corner point of class p, Pos represents the positive anchor point, and Neg represents the negative anchor point. The loss of classification confidence is defined based on the positive and negative anchor points, while the loss of the border or corner information is defined based on the positive anchor point.

[0054] in, Represents the predicted offset relative to the anchor, which can be expressed by the following formula:

[0055]

[0056] l can represent the location of the bounding box, and l can be obtained by center (l cx ,l cy ) and size (l w ,l h ) is used to represent, l cx represents the horizontal coordinate of the center point, l cy Represents the ordinate of the center point, l w Represents the width, l h Represents length.

[0057] L corner It can be calculated by the following formula:

[0058]

[0059] N is the total number of matched corner points, L conf (c (p) ) is the softmax loss value of the predicted corner point, For forecast data With real data The smooth loss between the predicted data g and the real data g* can be calculated by the following formula:

[0060]

[0061] It can be calculated by the following formula:

[0062]

[0063] Can represent predicted corner points To the default corner point P (d) The offset of It can be expressed by the following formula:

[0064]

[0065] For numerical stability, a normalized two-dimensional vector Expressed in the following redundant form:

[0066]

[0067] in, Unordered corner points The direction vector pointing to the center of the marker.

[0068] L mask It can be calculated by the following formula:

[0069]

[0070] M represents the prediction mask, Represents the true mask.

[0071] In which, given an input image of size h×w, the feature X0 is extracted using the MobileNet backbone encoder and residual blocks with spatial dimension h / 8×w / 8. MobileNet is more efficient than VGG-16 because it uses depthwise separable convolutions, which reduces computational complexity while retaining feature extraction capabilities. Since the fiducial markers consist of simple small shapes, mid-level features are extracted before high-level object detection. The two-channel mask M, of size h / 8×w / 8×2, represents the segmentation of the marker and background. X0 has several convolutional layers. The backbone feature X0 and the mask M are then concatenated as the intermediate feature X1. Two predictors are used to predict bounding boxes and corner points, respectively.

[0072] In some embodiments, the area where the mark is located is a target area. When there are multiple target areas, the confidence of each target area in the multiple target areas is obtained to obtain multiple confidences; the target area corresponding to the maximum confidence among the multiple confidences is obtained as the area where the mark is located. In this way, the higher the confidence, the more consistent the obtained target area is with the area contained in the actual frame of the mark. Therefore, by obtaining the target area with the maximum confidence as the area where the mark is located, the accuracy of the mark determination can be improved.

[0073] In other embodiments, the area where the mark is located is a target area. When there are multiple target areas, the confidence of each target area in the multiple target areas is obtained; based on the confidence of each target area, it is determined whether there are a target number of candidate areas in the multiple target areas, the confidence of the candidate areas is greater than the preset confidence, and the target number is at least two; if there are, the mean of the coordinate information of the target number of candidate areas is obtained as the target coordinate information; the area corresponding to the target coordinate information is obtained as the area where the mark is located. In this way, when there are multiple large confidences at the same time, the mean of the coordinate information of the target areas corresponding to the multiple large confidences is obtained, and the mean coordinate information is used as the area where the mark is located, which can prevent the computer device from switching back and forth between multiple candidate areas, resulting in the inability to accurately determine the area where the mark is located, and then causing inaccurate subsequent recognition of the mark.

[0074] Step S240: extracting a plurality of image feature points in the image corresponding to the area where the mark is located based on a pre-trained key point detection model.

[0075] In this embodiment, a pre-trained key point detection model can also be used to extract multiple image feature points in the image corresponding to the area where the mark is located, that is, Figure 3 Stage 2 shown in Figure 2. The key point detection model is also predicted based on a single-scale anchor predictor. Given an input image of size w×w, the anchor points are arranged in a grid of size w / 8×w / 8. On the other hand, the template predictor takes the intermediate features as input and estimates A single-scale method with 2×2 anchors is used. The specific training process can be found in the above-mentioned embodiment, which will not be described in detail here.

[0076] Among them, the total loss function L in the key point detection stage is detect It can be calculated by the following formula:

[0077] L detect =L keypoint +L template +L mask

[0078] Among them, L mask represents the aforementioned mask loss, L keypoint represents the loss of the single-scale anchor predictor, L template represents the loss of the template predictor.

[0079] L template and L keypoint They can be calculated by the following formulas:

[0080]

[0081]

[0082] in, and According to c (K) and c (T) Calculated positioning.

[0083] Step S250: Acquire the types of the plurality of image feature points, wherein the number of the types is determined based on the mark type of the mark.

[0084] In this embodiment, after extracting multiple image feature points, the type of each image feature point in the multiple image feature points can be further determined, wherein the type of the image feature point is determined based on the tag type of the tag, and the type of the image feature point can include only coded feature points, non-coded feature points, or visible feature points and invisible feature points. Exemplarily, when the tag type of the tag is TopoTag, the types of the multiple image feature points include coded feature points and non-coded feature points; when the tag type of the tag is RuneTag, the types of the multiple image feature points include visible feature points and invisible feature points.

[0085] Step S260: Based on the multiple image feature points and the types of the image feature points, the marker is identified to obtain an identification result, and the identification result is used to track and locate the marker.

[0086] In some embodiments, see Figure 4 , step S260 may include the following steps:

[0087] Step S261: sorting the plurality of image feature points based on a predefined template corresponding to the mark type of the mark to obtain a plurality of ordered feature points.

[0088] In this embodiment, since the multiple image feature points extracted by the key point detection model are disordered, if the identity document (ID) of the tag is to be identified, the tag ID may not be identified based on the disordered image feature points, or the tag ID may be misidentified. Therefore, multiple image feature points can be sorted to obtain multiple ordered feature points. Among them, sorting multiple image feature points can obtain a predefined template corresponding to the tag type of the tag as a target sorting template; according to the arrangement order of the key points in the target sorting template, a corresponding number is assigned to each of the multiple image feature points, that is, the multiple image feature points are sorted. Exemplarily, the three tag types of AprilTag, TopoTag and RuneTag correspond to a predefined template respectively. The predefined template can be pre-set or adjusted according to the actual application scenario. This embodiment does not limit this.

[0089] Step S262: obtaining a recognition result of the mark based on the type of each ordered feature point in the plurality of ordered feature points and the coordinate position information of the plurality of ordered feature points in the mark image.

[0090] Specifically, based on the type of each ordered feature point in the plurality of ordered feature points, the encoding value of each ordered feature point is obtained; according to the encoding value of each ordered feature point, the tag number (i.e., tag ID) corresponding to the tag is determined; through the inverse homography matrix, the coordinate position information of the plurality of ordered feature points in the tag image is obtained. According to the coordinate position information, the pose information of the tag is determined; based on the pose information and the tag number, the recognition result is generated. It can be understood that the pixel coordinates of the plurality of ordered feature points in the tag image are first obtained, and the plane where the image feature points are located is the world coordinate XY plane, and the origin of the world coordinate is determined in the plane; then the coordinate information of the image feature points in the world coordinate system in the camera coordinate system and the coordinate information of the camera in the world coordinate system can be obtained, and the pose information of the tag can be determined. Among them, the pose information includes six degrees of freedom information, i.e., coordinate position information and angle position information. In this way, the tag can be tracked and positioned in real time.

[0091] In this embodiment, by obtaining the corner points in the marked image to determine the area where the mark is located, it is possible to more accurately and quickly determine the area where the mark is located; based on this, more and denser image feature points can be extracted to achieve higher posture accuracy; and the mark is identified in combination with the type of image feature points, which also greatly improves the robustness of mark recognition.

[0092] Please refer to Figure 5 , Figure 5A flowchart of a marking recognition method provided in another embodiment of the present application is shown below. Figure 5 The tag recognition method provided in the embodiment of the present application is described in detail. The tag recognition method may include the following steps:

[0093] Step S401: Acquire a marker image including markers in a real environment.

[0094] Step S402: Obtaining a target number of corner points in the marked image.

[0095] Step S403: Based on the target number of corner points, determine the area where the mark is located from the mark image.

[0096] Step S404: extracting a plurality of image feature points in the image corresponding to the area where the mark is located based on a pre-trained key point detection model.

[0097] Step S405: Acquire the types of the plurality of image feature points, wherein the number of the types is determined based on the mark type of the mark.

[0098] In this embodiment, the specific implementation of steps S401 to S405 can refer to the contents of the above embodiments, which will not be described in detail here.

[0099] Step S406: According to a preset acquisition rule, a plurality of key feature points are acquired from the plurality of image feature points, and the local patterns corresponding to the key feature points include at least two types.

[0100] In this embodiment, the mark is composed of multiple local patterns. When designing the mark, the direction of the mark can be defined by introducing specially designed key points, and the local patterns corresponding to the key feature points include at least two. Based on this, the mark does not need to meet the rotational asymmetry in the related art, and the direction of the mark can be quickly determined by the key feature points. Optionally, the key feature points for defining the direction of the mark can be obtained from multiple image feature points through preset acquisition rules. For example, Figure 6 The image feature points P1, P2, P3, and P4 in are the multiple key feature points obtained. The key feature points P1, P2, and P3 are all the same local pattern, and the key feature point P4 is another local pattern.

[0101] Step S407: determining the direction of the mark according to the local pattern of each key feature point in the plurality of key feature points.

[0102] In practical applications, the marker in the marker image may not be in the positive direction due to different viewing angles of the marker image. It may be rotated by a certain angle relative to the preset standard direction, such as Figure 7 As shown, relative Figure 6 The mark located in the preset standard direction is rotated 90 degrees counterclockwise. Therefore, after obtaining multiple key feature points, the direction of the mark can be determined according to the local pattern of each key feature point.

[0103] Step S408: Determine whether the direction of the mark meets the preset standard direction.

[0104] Furthermore, it is determined whether the direction of the mark meets the preset standard direction, that is, whether the mark is placed in the forward direction. Specifically, the types of all key feature points can be sorted in a clockwise direction from the key feature point in the upper left corner to obtain the type sequence of the key feature points; it is determined whether the type sequence of the key feature points is consistent with the preset type sequence. If they are consistent, it is determined that the direction of the mark meets the preset standard direction; if not, it is determined that the direction of the mark does not meet the preset standard direction.

[0105] Step S409: If it is in compliance, executing the step of sorting the plurality of image feature points based on the predefined template corresponding to the mark type of the mark to obtain a plurality of ordered feature points.

[0106] Step S410: If not, the mark is rotated to the preset standard direction, and for the rotated mark, the step of sorting the multiple image feature points based on the predefined template corresponding to the mark type of the mark is performed to obtain multiple ordered feature points.

[0107] Based on this, if the direction of the mark does not conform to the preset standard direction, the mark is rotated to the preset standard direction, and the multiple image feature points identified in the area where the mark is located are sorted to obtain multiple ordered feature points.

[0108] Step S411: obtaining a recognition result of the mark based on the type of each ordered feature point in the plurality of ordered feature points and the coordinate position information of the plurality of ordered feature points in the mark image.

[0109] In this embodiment, the specific implementation of step S411 can refer to the content of the above-mentioned embodiment, which will not be repeated here.

[0110] In this embodiment, the direction of the mark can be determined by identifying the key feature points introduced to characterize the direction and the border of the mark. Compared with the method in the related art that can only identify the direction by a large number of bit conversions, this is more convenient and also improves the efficiency of mark recognition; and reduces the possibility of confusion with environmental elements, thereby improving the accuracy of mark recognition.

[0111] Please refer to Figure 8 , Figure 8A flowchart of a marking recognition method provided in yet another embodiment of the present application. Figure 8 The tag recognition method provided in the embodiment of the present application is described in detail. The tag recognition method may include the following steps:

[0112] Step S510: In response to the mark making instruction, display prompt information, multiple mark types and multiple local patterns, the local patterns are used to form the mark, and the prompt information is used to prompt the selection of the mark type and the local pattern.

[0113] In this embodiment, the reference mark can be designed by the user. The user can input a mark making instruction, and the computer device responds to the mark making instruction by displaying multiple mark types, multiple local patterns, and prompt information, and the prompt information is used to prompt the user to select the type of mark to be generated and select the local pattern used to generate the mark of the type. The type of mark can include but is not limited to Figure 2 The 5 types shown in the figure can have multiple colors of local patterns, not limited to black and white. In addition, local patterns can also include local patterns of multiple categories. The shapes and background colors of local patterns of the same category are the same, but the internal symbols of the local patterns can be different. The shapes and background colors of local patterns of different categories are different. For details, please refer to Fig. 9 There are various types of local patterns shown, including class1, class2, class3, ..., class C.

[0114] In some embodiments, the prompt information may also include prompt information for selecting key feature points, that is, prompting the user to select at least two different local patterns as local patterns for representing the direction of the mark. In this way, the direction of the mark can be more conveniently identified during subsequent mark recognition.

[0115] Step S520: In response to the confirmation instruction, obtaining the type of the mark carried in the confirmation instruction as the designated mark type, and obtaining the local pattern carried in the confirmation instruction as the designated local pattern.

[0116] Based on this, after the user selects the type of mark and the local pattern that constitutes the mark, a confirmation instruction can be input. Correspondingly, the computer device responds to the confirmation instruction, obtains the type of mark carried in the confirmation instruction as the specified mark type, and obtains the local pattern carried in the confirmation instruction as the specified local pattern.

[0117] Step S530: Based on the designated local pattern, generate a mark corresponding to the designated mark type.

[0118] In this embodiment, after obtaining the specified local pattern and the specified mark type, the computer device randomly arranges the local pattern according to the format of the specified mark type to generate a mark corresponding to the specified mark type; optionally, the local pattern can also be arranged and combined according to the specified arrangement and combination method to generate a mark corresponding to the specified mark type, which is not limited in this embodiment.

[0119] Step S540: Acquire a marker image including markers in the real environment.

[0120] Step S550: Obtaining the target number of corner points in the marked image.

[0121] Step S560: Based on the target number of corner points, determine the area where the mark is located from the mark image.

[0122] Step S570: extracting a plurality of image feature points in the image corresponding to the area where the mark is located based on a pre-trained key point detection model.

[0123] Step S580: Acquire the types of the plurality of image feature points, wherein the number of the types is determined based on the mark type of the mark.

[0124] Step S590: Based on the multiple image feature points and the types of the image feature points, the mark is identified to obtain a recognition result, and the recognition result is used to track and locate the mark.

[0125] In this embodiment, the specific implementation of steps S540 to S590 can refer to the contents of the aforementioned embodiments and will not be described in detail here.

[0126] In this embodiment, a local mode of mark customization is provided to the user, that is, the user can select the local pattern and mark type he needs, so that the generated new mark can be more in line with the user's aesthetics and needs; and the local pattern that constitutes the mark can include a variety of colors and shapes, which greatly reduces the possibility of confusion with environmental elements, thereby improving the accuracy and robustness of mark recognition.

[0127] Please refer to Fig.10 , Fig.10 A flowchart of a marking recognition method provided in yet another embodiment of the present application is shown below. Fig.10 The tag recognition method provided in the embodiment of the present application is described in detail. The tag recognition method may include the following steps:

[0128] Step S610: Acquire a marker image including markers in the real environment.

[0129] Step S620: Obtain the target number of corner points in the marked image.

[0130] Step S630: Based on the target number of corner points, determine the area where the mark is located from the mark image.

[0131] Step S640: Obtain the tag type of the tag as the target tag type.

[0132] Step S650: Obtain a key point detection model corresponding to the target mark type as a target detection model.

[0133] In this embodiment, the specific implementation of steps S610 to S650 can refer to the contents of the aforementioned embodiments and will not be described in detail here.

[0134] Step S660: Based on the target detection model, extract multiple image feature points in the image corresponding to the area where the mark is located.

[0135] In some implementations, if the target tag type is a first tag type, based on the target detection model, a plurality of visible feature points and a plurality of invisible feature points in the image corresponding to the region where the tag is located are extracted as the plurality of image feature points. For example, for RuneTag type tags, see Fig.11 In the related art, image feature points are only defined at visible points. Therefore, when the quality of the marked image is poor, it will be difficult to sort multiple image feature points, which will affect the recognition of the mark. In this embodiment, image feature points are defined at both visible and invisible points. Fig.12 , black points represent visible feature points, and gray points represent invisible feature points. In this way, the rules can be used for sorting, making it easier to sort multiple image feature points, thereby improving the accuracy of subsequent mark recognition.

[0136] In some other embodiments, if the target tag type is the second tag type, based on the target detection model, a plurality of first coded bit feature points, a plurality of second coded bit feature points, and non-coded bit feature points in the image corresponding to the area where the tag is located are extracted as the plurality of image feature points. For example, for a TopoTag type tag, in the related art, image feature points only include coded feature points, where coded feature points can represent "0" or "1", see Fig.13 As shown. In this embodiment, the image feature points can be divided into three categories, the first coding bit feature point "0", the second coding bit feature point "1" and the non-coding bit feature point, wherein the non-coding bit feature point can be defined in the area without local pattern. For details, please refer to Fig.14In this way, more dense image feature points can be obtained to facilitate sorting of the image feature points, thereby improving the accuracy of marker recognition.

[0137] In some other embodiments, if the target tag type is the third tag type, based on the target detection model, a preset number of image feature points corresponding to the preset size of the tag are extracted as multiple image feature points. For example, for the AprilTag type tag, in the related art, refer to Fig.15 , only the image feature points are defined at the four corners of the boundary of the mark, and the feature points inside the mark are not encoded. In this embodiment, more dense image feature points in the image inside the mark can be extracted and the image feature points can be encoded and defined. Please refer to Fig.16 , it can be defined that the image feature points in the white area represent the coding bit "1", and the image feature points in the black area represent the coding bit "0". In this way, more dense image feature points can be obtained in addition to the four image feature points on the boundary, thereby improving the accuracy of identifying the mark based on the denser image feature points.

[0138] Step S670: Acquire the types of the plurality of image feature points, wherein the number of the types is determined based on the mark type of the mark.

[0139] Step S680: Based on the multiple image feature points and the types of the image feature points, the mark is identified to obtain a recognition result, and the recognition result is used to track and locate the mark.

[0140] In this embodiment, the specific implementation of steps S670 to S680 can refer to the contents of the aforementioned embodiments and will not be repeated here.

[0141] In this embodiment, the marker type of the marker is determined to determine the target detection model corresponding to the marker type, and then more image feature points are detected by the target detection model. In this way, for markers of different marker types, a pre-trained detection model can be used to extract more dense image feature points from the marker, and then more dense and arranged image feature points can be obtained, so as to achieve more accurate identity recognition and position information recognition of the marker, and improve the accuracy of marker tracking and positioning.

[0142] Please refer to Fig.17 , which shows a structural block diagram of a marker recognition device 700 provided by an embodiment of the present application. The device 700 may include: an image acquisition module 710, a corner point acquisition module 720, a region determination module 730, a feature point extraction module 740, a category acquisition module 750 and a marker recognition module 760.

[0143] The image acquisition module 710 is used to acquire a marked image including marks in a real environment;

[0144] The corner point acquisition module 720 is used to acquire the corner points of the target number in the marked image;

[0145] The region determination module 730 is used to determine the region where the mark is located from the mark image based on the corner points of the target number;

[0146] The feature point extraction module 740 is used to extract a plurality of image feature points in the image corresponding to the area where the mark is located based on a pre-trained key point detection model;

[0147] The category acquisition module 750 is used to acquire the categories of the plurality of image feature points, wherein the number of the categories is determined based on the mark type of the mark;

[0148] The marker recognition module 760 is used to recognize the marker based on the multiple image feature points and the types of the image feature points to obtain a recognition result, and the recognition result is used to track and locate the marker.

[0149] In some embodiments, the marker recognition module 760 may include: a sorting unit and a recognition unit. The sorting unit may be used to sort the multiple image feature points based on a predefined template corresponding to the marker type of the marker to obtain multiple ordered feature points. The recognition unit may be used to obtain a recognition result of the marker based on the type of each ordered feature point in the multiple ordered feature points and the coordinate position information of the multiple ordered feature points in the marker image.

[0150] In this manner, the recognition unit may include: a coding acquisition subunit, a number acquisition subunit, a coordinate acquisition subunit, a posture acquisition subunit and a recognition subunit. Among them, the coding acquisition subunit can be used to obtain the coding value of each ordered feature point based on the type of each ordered feature point in the multiple ordered feature points. The number acquisition subunit can be used to determine the tag number corresponding to the tag according to the coding value of each ordered feature point. The coordinate acquisition subunit can be used to obtain the coordinate position information of the multiple ordered feature points in the tag image through an inverse homography matrix. The posture acquisition subunit can be used to determine the posture information of the tag according to the coordinate position information. The recognition subunit can be used to generate the recognition result based on the posture information and the tag number.

[0151] In some embodiments, the types of the image feature points correspond to the types of the local patterns constituting the mark, and the mark recognition device 700 may further include: a key point acquisition module, a direction determination module, and a direction judgment module. Among them, the key point acquisition module can be used to obtain multiple key feature points from the multiple image feature points according to a preset acquisition rule before sorting the multiple image feature points based on a predefined template corresponding to the mark type of the mark to obtain multiple ordered feature points, and the local patterns corresponding to the key feature points include at least two types. The direction determination module can be used to determine the direction of the mark according to the local pattern of each key feature point in the multiple key feature points. The direction judgment module can be specifically used to determine whether the marking method meets the preset standard direction; if it meets, the step of sorting the multiple image feature points based on the predefined template corresponding to the mark type of the mark to obtain multiple ordered feature points is executed; if it does not meet, the mark is rotated to the preset standard direction, and for the rotated mark, the step of sorting the multiple image feature points based on the predefined template corresponding to the mark type of the mark to obtain multiple ordered feature points is executed.

[0152] In this manner, the mark recognition device 700 may further include: a display module, a mark information acquisition module, and a mark generation module. The display module may be used to display prompt information, multiple mark types, and multiple local patterns in response to a mark making instruction before acquiring a mark image containing a mark in a real environment, wherein the local pattern is used to constitute the mark, and the prompt information is used to prompt the selection of the mark type and the local pattern. The mark information acquisition module may be used to acquire the type of the mark carried in the confirmation instruction as a specified mark type in response to a confirmation instruction, and acquire the local pattern carried in the confirmation instruction as a specified local pattern. The mark generation module may be used to generate a mark corresponding to the specified mark type based on the specified local pattern.

[0153] In some embodiments, the feature point extraction module 740 may include: a type acquisition unit, a model acquisition unit, and a feature point extraction unit. The type acquisition unit may be used to acquire the tag type of the tag as the target tag type. The model acquisition unit may be used to acquire a key point detection model corresponding to the target tag type as the target detection model. The feature point extraction unit may be used to extract multiple image feature points in the image corresponding to the area where the tag is located based on the target detection model.

[0154] In this manner, the feature point extraction unit can be specifically used to extract multiple visible feature points and multiple invisible feature points in the image corresponding to the area where the mark is located, as the multiple image feature points, based on the target detection model, if the target mark type is the first mark type.

[0155] In this manner, the feature point extraction unit can also be specifically used to extract, if the target mark type is the second mark type, a plurality of first coding bit feature points, a plurality of second coding bit feature points and non-coding bit feature points in the image corresponding to the area where the mark is located, based on the target detection model, as the plurality of image feature points.

[0156] In some embodiments, the region where the mark is located is a target region. When there are multiple target regions, the region determination module 730 may include: a confidence acquisition unit and a region determination unit. The confidence acquisition unit may be used to acquire the confidence of each target region in the multiple target regions to obtain multiple confidences. The region determination unit may be used to acquire the target region corresponding to the maximum confidence among the multiple confidences as the region where the mark is located.

[0157] In other embodiments, the area where the mark is located is a target area. When the number of the target areas is multiple, the area determination module 730 may include: a confidence acquisition unit, a judgment unit, a coordinate information acquisition unit, and an area determination unit. Among them, the confidence acquisition unit can be used to obtain the confidence of each target area in the multiple target areas. The judgment unit can be used to determine whether there are a target number of to-be-selected areas in the multiple target areas based on the confidence of each target area, the confidence of the to-be-selected areas is greater than the preset confidence, and the number of targets is at least two. The coordinate information acquisition unit can be used to obtain the mean of the coordinate information of the target number of to-be-selected areas as the target coordinate information if there are a target number of to-be-selected areas in the multiple target areas. The area determination unit can be used to obtain the area corresponding to the target coordinate information as the area where the mark is located.

[0158] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0159] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0160] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0161] In summary, in the solution provided by the embodiment of the present application, a marked image containing marks in a real environment is obtained; corner points of a target number are obtained in the marked image; based on the corner points of the target number, the area where the mark is located is determined from the marked image; based on a pre-trained key point detection model, multiple image feature points in the image corresponding to the area where the mark is located are extracted; the types of multiple image feature points are obtained, and the number of types is determined based on the mark type of the mark; based on multiple image feature points and the types of image feature points, the mark is identified to obtain a recognition result, and the recognition result is used to track and locate the mark. In this way, by obtaining the corner points in the marked image to determine the area where the mark is located, it is possible to more accurately and quickly determine the area where the mark is located; based on this, more image feature points can be extracted to achieve higher posture accuracy; and the mark is identified in combination with the type of image feature points, which also greatly improves the robustness of mark recognition.

[0162] The following will be combined Fig.18 A computer device provided by the present application is described.

[0163] Reference Fig.18 , Fig.18 The structural block diagram of a computer device 800 provided in an embodiment of the present application is shown. The tag recognition method provided in an embodiment of the present application can be executed by the computer device 800. The computer device 800 can be a device capable of running an application program.

[0164] The computer device 800 in the embodiment of the present application may include one or more of the following components: a processor 801, a memory 802, and one or more applications, wherein the one or more applications may be stored in the memory 802 and configured to be executed by one or more processors 801, and the one or more programs are configured to execute the method as described in the aforementioned method embodiment.

[0165] The processor 801 may include one or more processing cores. The processor 801 uses various interfaces and lines to connect various parts of the entire computer device 800, and executes various functions and processes data of the computer device 800 by running or executing instructions, programs, code sets or instruction sets stored in the memory 802, and calling data stored in the memory 802. Optionally, the processor 801 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 801 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem can also be integrated into the processor 801 and implemented separately through a communication chip.

[0166] The memory 802 may include a random access memory (RAM) or a read-only memory (ROM). The memory 802 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as the various corresponding relationships described above) created by the computer device 800 during use.

[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0168] In several embodiments provided in the present application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or modules, which may be electrical, mechanical or other forms.

[0169] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0170] Please refer to Fig.19 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 900 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0171] The computer readable storage medium 900 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer readable storage medium 900 has storage space for program code 910 that performs any method step of the above method. These program codes can be read from or written to one or more computer program products. The program code 910 can be compressed, for example, in an appropriate form.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for identifying a mark, It is characterized in that The method comprises: Obtaining a labeled image including a marker in a real environment; Obtaining corner points of a target number in the marked image; Based on the corner points of the target number, determining the area where the mark is located from the mark image; Extracting a plurality of image feature points in the image corresponding to the area where the mark is located based on a pre-trained key point detection model; Acquire types of the plurality of image feature points, wherein the number of the types is determined based on a mark type of the mark; Based on the multiple image feature points and the types of the image feature points, the marker is identified to obtain a recognition result, and the recognition result is used to track and locate the marker.

2. The method according to claim 1, It is characterized in that The identifying the mark based on the multiple image feature points and the types of the image feature points to obtain the identification result includes: sorting the plurality of image feature points based on a predefined template corresponding to the mark type of the mark to obtain a plurality of ordered feature points; Based on the type of each ordered feature point in the plurality of ordered feature points and the coordinate position information of the plurality of ordered feature points in the mark image, a recognition result of recognizing the mark is obtained.

3. The method according to claim 2, It is characterized in that The obtaining of a recognition result of the mark based on the type of each ordered feature point in the plurality of ordered feature points and the coordinate position information of the plurality of ordered feature points in the mark image comprises: Based on the type of each ordered feature point in the plurality of ordered feature points, obtaining a code value of each ordered feature point; Determine the mark number corresponding to the mark according to the code value of each ordered feature point; Obtaining coordinate position information of the plurality of ordered feature points in the marked image through an inverse homography matrix; Determining the position information of the marker according to the coordinate position information; The recognition result is generated based on the posture information and the tag number.

4. The method according to claim 2, It is characterized in that The types of the image feature points correspond one-to-one to the types of the local patterns constituting the mark. Before sorting the plurality of image feature points based on a predefined template corresponding to the mark type of the mark to obtain a plurality of ordered feature points, the method further includes: According to a preset acquisition rule, a plurality of key feature points are acquired from the plurality of image feature points, wherein the local patterns corresponding to the key feature points include at least two types; determining a direction of the mark according to a local pattern of each key feature point among the plurality of key feature points; Determining whether the marking method complies with a preset standard direction; If yes, then executing the step of sorting the plurality of image feature points based on a predefined template corresponding to the mark type of the mark to obtain a plurality of ordered feature points; If it does not meet the requirements, the mark is rotated to the preset standard direction, and for the rotated mark, the step of sorting the multiple image feature points based on the predefined template corresponding to the mark type of the mark is performed to obtain multiple ordered feature points.

5. The method according to claim 4, It is characterized in that Before acquiring the marked image containing the marks in the real environment, the method further includes: In response to a mark making instruction, displaying prompt information, a plurality of mark types and a plurality of partial patterns, wherein the partial patterns are used to form the mark, and the prompt information is used to prompt selection of the mark type and the partial pattern; In response to a confirmation instruction, obtaining a type of a mark carried in the confirmation instruction as a designated mark type, and obtaining a local pattern carried in the confirmation instruction as a designated local pattern; Based on the designated local pattern, a mark corresponding to the designated mark type is generated.

6. The method according to claim 1, It is characterized in that The extracting of a plurality of image feature points in the image corresponding to the area where the mark is located based on the pre-trained key point detection model includes: Obtaining the tag type of the tag as the target tag type; Acquire a key point detection model corresponding to the target mark type as a target detection model; Based on the target detection model, a plurality of image feature points in the image corresponding to the area where the mark is located are extracted.

7. The method according to claim 6, It is characterized in that The extracting, based on the target detection model, a plurality of image feature points in the image corresponding to the area where the mark is located, comprises: If the target mark type is the first mark type, based on the target detection model, a plurality of visible feature points and a plurality of invisible feature points in the image corresponding to the area where the mark is located are extracted as the plurality of image feature points.

8. The method according to claim 6, It is characterized in that The extracting, based on the target detection model, a plurality of image feature points in the image corresponding to the area where the mark is located, comprises: If the target mark type is the second mark type, based on the target detection model, multiple first coding bit feature points, multiple second coding bit feature points and non-coding bit feature points in the image corresponding to the area where the mark is located are extracted as the multiple image feature points.

9. The method according to any one of claims 1 to 8, It is characterized in that The area where the mark is located is the target area. When there are multiple target areas, determining the area where the mark is located based on the corner points of the target number includes: Obtaining the confidence of each target area in the multiple target areas to obtain multiple confidences; A target area corresponding to the maximum confidence level among the multiple confidence levels is obtained as the area where the mark is located.

10. A marking recognition device, It is characterized in that The device comprises: An image acquisition module, used for acquiring a marked image containing marks in a real environment; A corner point acquisition module, used to acquire the corner points of the target number in the marked image; A region determination module, used for determining the region where the mark is located from the mark image based on the corner points of the target number; A feature point extraction module, used to extract a plurality of image feature points in the image corresponding to the area where the mark is located based on a pre-trained key point detection model; A category acquisition module, used for acquiring categories of the plurality of image feature points, wherein the number of the categories is determined based on the mark type of the mark; The marker recognition module is used to recognize the marker based on the multiple image feature points and the types of the image feature points to obtain a recognition result, and the recognition result is used to track and locate the marker.

Images