Marker and method for identifying marker
By introducing submarks with feature points into markers, the difficulty of identification and tracking caused by the lack of feature points in the prior art is solved, and more efficient and stable marker recognition and tracking effects are achieved, which are suitable for augmented reality and other Technical field.
Patent Information
- Application Number
- CN201910234813.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-03-26
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2039-03-26
AI Technical Summary
In the absence of feature points, existing markers are difficult to effectively identify and track objects, especially in technical fields such as augmented reality, which affects the superposition effect of virtual and reality and the stability of identification and tracking.
A marker is designed, including a background and a plurality of sub-markers distributed in the background. The sub-markers include a starting sub-marker and a feature sub-marker. The starting sub-marker has at least two feature points. The feature sub-marker may have one or more By identifying the information of these sub-markers, identifying the identity information of the target marker, improving the accuracy and stability of identification and tracking.
实现了更快、更准确的标记物识别,适用于自动化和批量化生产,能够根据需求调整标记物大小,增加或减少子标记数量,显著提高了标记物的识别效果和识别追踪的稳定性。
Smart Images

Figure CN111754570B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of markers, and in particular to a marker and a method for identifying a marker. Background Art
[0002] In computer vision applications, markers usually play a core role. Markers can be widely used in various scenarios to produce corresponding effects. In traditional marker design schemes, corresponding solutions can be proposed according to different application requirements. For example, some markers focus on high-precision restoration of the camera's relative posture to the marker, while some markers focus on optimizing the detection speed or optimizing the impact of occlusion on the detection process, etc. Summary of the invention
[0003] An embodiment of the present application provides a marker recognition method, including: acquiring an image containing a target marker, the target marker including a background and multiple sub-markers distributed on the background, the multiple sub-markers including a starting sub-marker and one or more characteristic sub-markers; identifying the starting sub-marker of the target marker in the image, and acquiring first information of the starting sub-marker, the starting sub-marker including at least two characteristic points; identifying the characteristic sub-marker of the target marker in the image, and acquiring second information of the characteristic sub-marker, the characteristic sub-marker including one characteristic point or no characteristic point; determining the identity information of the target marker according to the first information and the second information.
[0004] The present application also provides a marker for image capture by an image sensor for identification and tracking, the marker including a background and multiple sub-markers distributed in the background; the multiple sub-markers include a starting sub-marker and one or more feature sub-markers, the starting sub-marker includes at least two feature points, and each feature sub-marker includes one feature point or no feature point.
[0005] The present application also provides a marker identification device, including: an acquisition module, used to acquire an image containing a target marker, the target marker includes a background, and multiple sub-markers distributed on the background, the multiple sub-markers include a starting sub-marker and one or more characteristic sub-markers; a first identification module, used to identify the starting sub-marker of the target marker in the image, and obtain first information of the starting sub-marker, the starting sub-marker includes at least two characteristic points; a second identification module, used to identify the characteristic sub-marker of the target marker in the image, and obtain second information of the characteristic sub-marker, the characteristic sub-marker includes one characteristic point or does not include a characteristic point; a determination module, used to determine the identity information of the target marker based on the first information and the second information.
[0006] The present application also provides a terminal device, comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured for a method of identifying a marker.
[0007] The present application also provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute a method for identifying a marker.
[0008] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work. As shown in the accompanying drawings, the above and other purposes, features and advantages of the present application will be clearer. The same reference numerals indicate the same parts in all the drawings. The drawings are not deliberately scaled to the actual size, and the focus is on showing the subject matter of the present application.
[0010] Figure 1 This is a schematic diagram of an application scenario of a marker in AR.
[0011] Figure 2 Schematic diagram of marker types in the prior art.
[0012] Figure 3 A schematic diagram of a marker provided in an embodiment of the present application.
[0013] Figure 4A A schematic diagram of the recognition sequence of characteristic sub-markers according to an embodiment of the present application.
[0014] Figure 4B This is a schematic diagram of the recognition sequence of characteristic sub-markers according to another embodiment of the present application.
[0015] Figure 4C-4D Schematic diagram of markers with different identities provided in the embodiments of the present application.
[0016] Figure 5 A schematic diagram of another marker provided in an embodiment of the present application.
[0017] Figure 6 A schematic diagram of another marker provided in an embodiment of the present application.
[0018] Figure 7 A schematic diagram of another marker provided in an embodiment of the present application.
[0019] Figure 8 A flow chart of a method for identifying a marker provided in an embodiment of the present application.
[0020] Fig. 9 A schematic diagram of calculating the arrangement relationship of characteristic sub-markers provided in an embodiment of the present application.
[0021] Fig.10 A schematic diagram of calculating a characteristic sub-marker arrangement relationship provided in another embodiment of the present application.
[0022] Fig.11 This is a block diagram of a device for identifying a marker according to an embodiment of the present application.
[0023] Fig.12 It is a block diagram of a terminal device according to an embodiment of the present application for executing the method for identifying a marker according to an embodiment of the present application.
[0024] Fig.13 A schematic diagram of an interactive device provided in an embodiment of the present application.
[0025] Fig.14 A schematic diagram of a tracking marker provided in an embodiment of the present application.
[0026] Fig.15 A schematic diagram of a light source, a light guide, and a single marker provided in an embodiment of the present application.
[0027] Fig.16 A schematic diagram of a light source, a light guide, and multiple markers provided in an embodiment of the present application. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0029] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0030] Feature detection plays a vital role in applications such as object recognition and tracking. In a specific scene, the recognition and tracking of actual objects is generally achieved by calculating the feature points corresponding to the object. However, in many scenes, there is a lack of effective corresponding feature points. An effective way to solve this problem is to add markers to the scene to increase the features. For example, a marker with a two-dimensional pattern can be added to the environment so that the camera can detect it and obtain the posture information of the camera relative to the marker. This method can be applied to virtual reality (VR), augmented reality (AR), mixed reality (MR), positioning navigation, three-dimensional modeling and other technical fields.
[0031] Taking the field of augmented reality technology as an example, augmented reality technology is a technology that increases the user's perception of the real world through information provided by a computer system. It superimposes computer-generated virtual objects, scenes, or system prompt information and other content objects into real scenes to enhance or modify the perception of the real-world environment or data representing the real-world environment. In order to achieve the technical effect of superimposing virtual and real in the field of augmented reality, it is necessary to identify and track the real environment. Identification and tracking based on markers is a common form in AR systems and can be applied to various fields such as education, entertainment, commerce, and industry.
[0032] Figure 1 A diagram of an application scenario of a marker in AR in one embodiment is shown, and the application scenario includes a display system 10. The display system 10 includes: a terminal device 100 and a marker 200. The terminal device 100 may be a head-mounted display device, or a mobile device such as a mobile phone or a tablet; when the marker 200 is within the visual range of the image sensor of the terminal device 100, the terminal device 100 may capture an image containing the marker 200, and perform recognition and tracking operations on the image containing the marker 200 to obtain the identity information of the marker 200, as well as information such as the posture and position of the marker 200 relative to the terminal device 100. The terminal device 100 may generate corresponding virtual content based on information such as the posture and position of the marker 200 relative to the terminal device 100. For example, in Figure 1 In the embodiment, the head mounted display device 100 worn by the user scans the marker 200 in real time, and finally the virtual character 401 and the virtual animal 402 can be seen superimposed on the real environment.
[0033] Figure 2 Some existing marker types are shown, each with different detection technologies and identification methods, for example, Figure 2 To identify the markers corresponding to (a) and (b), first detect the circular outline, and identify the marker corresponding to (a) by the color of the ring inside the circle, and identify the marker corresponding to (b) by the fan-shaped area inside the circle; Figure 2 In the identification of the markers corresponding to (c) and (d), the black square border is first detected, and then the marker corresponding to (c) is identified through the image, and the marker corresponding to (d) is identified through the error correction binary code. There are many types of existing markers, each with its own advantages and disadvantages, and each is suitable for different application scenarios.
[0034] The embodiment of the present application provides a new marker with better recognition and tracking effect and high stability in real-time tracking.
[0035] Figure 3 The figure shows the form of a marker in an embodiment of the present application. The marker 20 has a frame 21. In some embodiments, the frame 21 is a square frame, so that the marker 20 is square in shape as a whole. The marker 20 includes a background 22 located in the frame 21 and a plurality of sub-markers 23 distributed in the background 22. When the marker 20 is captured by the image sensor, the color of the frame 21 in the acquired marker image is significantly different from the color of the background 22, and the color of the sub-markers 23 in the image is significantly different from the color of the background 22. In a specific embodiment, the border 21 is black, the background 22 is white, and the sub-mark 23 is black; in some embodiments, the color difference between the border 21, the background 22, and the sub-mark 23 observed by the human eye can be very small. For example, the areas corresponding to the border 21 and the sub-mark 23 are set to be reflective, and the areas corresponding to the background 22 are set to be non-reflective. An infrared camera is used to capture an image of the marker. When infrared light is directed at the marker, there will be a large color difference between the border 21 and the background 22 in the marker image captured by the infrared camera, and there will be a large color difference between the background 22 and the sub-mark 23.
[0036] In some embodiments, the sub-marker 23 includes a starting sub-marker 231 and characteristic sub-markers 232 and 233. The starting sub-marker 231 can be regarded as an identifier of the marker 20. The position of the starting sub-marker 231 can be used to obtain the starting calculation position of the marker 20. The arrangement of the characteristic sub-markers 232 and 233 in the marker 20 can represent the identity information of the marker 20.
[0037] The starting sub-marker 231 has at least two characteristic points 2311. In this embodiment, the starting sub-marker 231 has two characteristic points 2311, and the outline of the starting sub-marker 231 is two intersecting circular rings, which are roughly in the shape of "8". The two characteristic points 2311 are dots set in the starting sub-marker 231 and separated from each other. The color of the dots in the collected marker image is greatly different from the color of the starting sub-marker 231. In a specific embodiment, the starting sub-marker 231 is black, and the characteristic points 2311 are white dots.
[0038] The characteristic sub-marker 232 is annular and has a characteristic point 2321, which is a dot set in the characteristic sub-marker 232. The color of the dot in the collected marker image is significantly different from the color of the characteristic sub-marker 232. In a specific embodiment, the characteristic sub-marker 231 is black and the characteristic point 2321 is a white dot.
[0039] The characteristic sub-marker 233 is a circle, which is a black solid circle. It is not limited whether each characteristic sub-marker in the marker 20 contains information of characteristic points. For example, the characteristic sub-markers included can be all circular characteristic sub-markers 232, or all solid circle characteristic sub-markers 233, or a combination of the same number or different numbers of the two.
[0040] The positions of the start sub-marker 231 and the characteristic sub-markers 232 and 233 in the background 22 are relatively fixed. In a specific embodiment, the start sub-marker 231 is located at a corner of the background 22, and the characteristic sub-markers 232 and 233 are fixed relative to the start sub-marker 231. The start sub-marker 231 and the characteristic sub-markers 232 and 233 can be arranged in a preset number of rows and columns. Figure 3 For example, the starting sub-mark 231 is located at the upper left corner of the background 22, the characteristic sub-marks 232 and 233 are located to the right and below the starting sub-mark 231, and the starting sub-mark 231 and the characteristic sub-marks 232 and 233 are arranged in three rows, and the starting sub-mark 231 and the characteristic sub-marks 232 and 233 are arranged in the first row in two columns, and the remaining characteristic sub-marks 232 and 233 are arranged in the second and third rows in three columns, wherein the row and column spacing between every two adjacent characteristic sub-marks 232 and 233 is the same.
[0041] The characteristic sub-marker may have corresponding coding information. In one embodiment, the characteristic sub-marker may be binary coded. For example, the binary coding of the characteristic sub-marker 232 is "1", and the binary coding of the characteristic sub-marker 233 is "0". The characteristic sub-markers may be identified in a certain order to obtain the coding of each characteristic sub-marker, thereby obtaining the identity information of the marker. For example, the characteristic sub-markers may be identified in a clockwise order, such as Figure 4A As shown in FIG. 1 , the characteristic sub-marks are identified in sequence from numbers 1 to 7. For another example, according to Figure 4B The characteristic sub-markers are identified from top to bottom and from left to right according to the sequence numbers 1-7 in the table. Figure 4A The clockwise recognition order can be obtained Figure 4C The identity code of the marker in Figure 4D Different from Figure 4C The identity code of the marker is "0 110110"; according to Figure 4B The recognition order from top to bottom and from left to right can be obtained Figure 4C The marker identity code is "0011001" in Figure 4D Different from Figure 4C The identity code of the marker is "0101101".
[0042] In some embodiments, the number of feature points in the start sub-marker 231 is not limited. Figure 5 As shown, there are three characteristic points.
[0043] In some embodiments, the number of feature sub-markers is not limited Figure 3 The specific number can be designed according to the situation and requirements of the tracking scene.
[0044] In some embodiments, the outline of the sub-marker can be various, for example, it can be square, triangle, polygon, etc., and is not limited to the above-mentioned circle. Figure 6 As shown, the outer contours of the sub-markers are all square, and the feature points are also square in shape.
[0045] In some embodiments, the start sub-marker 231 can be located at any position of the background 22, and is not limited to the above-mentioned corners. Figure 7 As shown, the start sub-mark is located on the right side of the second row; in some embodiments, the arrangement of the characteristic sub-marks is not limited to the above-mentioned row and column arrangement, and the row and column spacing between each characteristic sub-mark can be the same or different.
[0046] In some embodiments, the start sub-marker 231 can be used to assist in obtaining the identity information of the marker. When the codes corresponding to the characteristic sub-markers of two or more markers are consistent, the identity information of the marker can be determined by identifying the number of characteristic points contained in the start sub-marker 231. For example, the codes corresponding to the characteristic sub-markers of two markers are both "0110001", but the start sub-marker of one marker contains 2 characteristic points and the start sub-marker of the other marker contains 3 characteristic points, so they can be distinguished. In some embodiments, the start sub-markers containing different numbers of characteristic points can correspond to different codes. For example, the code of the start sub-marker containing 2 characteristic points is 2, and the code of the start sub-marker containing 3 characteristic points is 3. Then the identity code of the marker can be a combination of the code of the start sub-marker and the code of the characteristic sub-marker. In other embodiments, the code of the start sub-marker is not limited to numbers, but can also be letters, symbols, etc. For example, X is used to represent the start sub-marker containing 2 characteristic points, and Y is used to represent the start sub-marker containing 3 characteristic points. The code of the characteristic sub-marker is not limited to binary code, and the embodiment of the present application does not limit the encoding method.
[0047] See also Figure 8 , an embodiment of the present application provides a method for identifying a marker, which is applied to an electronic device, and the method may include:
[0048] Step 110: Acquire an image containing target markers.
[0049] In some embodiments, the electronic device may include an image sensor and a processor. The image sensor may be any image sensing device capable of capturing an image of an object within its field of view, which is used to capture images of the marker. The image sensor may be configured to capture multiple images at different time points over a period of time. For example, when the marker moves within the field of view of the image sensor, the image sensor may capture images of the marker at different positions during the period of time. The image sensor may also obtain time information when capturing each image. The image sensor may also send the time information together with the image to the processor for further processing. The image sensor may communicate with the processor and send the image data to the processor, and the image sensor may also receive a command signal from the processor to set the parameters for capturing the image. The processor may be any suitable type of general or special purpose microprocessor, digital signal processor or microcontroller, which may be configured as a separate processor module dedicated to marker identification. The electronic device may also include any suitable type of memory for storing any type of information that the processor may need to operate, which may be configured to store one or more computer programs that can be executed by the processor, and may also be configured to store information and data used by the processor.
[0050] In some embodiments, the electronic device may further include an active lighting module for providing ambient light for image capture of the marker.
[0051] In some embodiments, the electronic device may pre-process the image containing the marker captured by the image sensor so as to distinguish the border, background, sub-markers of the marker, and the connected domains corresponding to the corresponding feature points from the image. As a specific embodiment, the captured image may be processed into a binary image, for example, the image may be binarized using a fixed threshold method or an adaptive threshold method.
[0052] As a specific implementation method, Figure 3 Taking the marker 20 shown as an example, after the captured image is binarized, the border 21 and sub-markers 231, 232, 232 of the marker in the image are processed as the first color, and the background 22 and feature points 2311, 2321 in the marker are processed as the second color; specifically, the first color can be black, and the second color can be white. In other words, the electronic device processes the parts of the marker that are in a surrounding relationship in sequence into having a color hierarchy, and forms a connected domain that is surrounded in sequence between the parts. Among them, the first color and the second color can be colors with a large difference in pixel values, such as the first color is black and the second color is white. Of course, in the binarized image, the distinction between the border, background, sub-markers and corresponding feature points can also be achieved through other methods such as contrast. The embodiment of the present application is mainly explained by taking the color hierarchy as an example.
[0053] Step 120: Identify the starting sub-marker of the target marker in the image, and obtain first information of the starting sub-marker.
[0054] In some embodiments, for an acquired image containing a target marker, the background of the target marker and a sub-marker located in the background are identified, and a sub-marker having at least two feature points is further identified among the identified sub-markers, and the sub-marker is determined as a starting sub-marker; after determining the starting sub-marker, first information of the starting sub-marker is acquired, and the first information includes at least the pixel coordinates of the starting sub-marker in the acquired image containing the target marker, the number of feature points included in the starting sub-marker, etc., wherein the pixel coordinates can be used to represent the position of the pixel point in the image.
[0055] In some embodiments, the electronic device may obtain the enclosing relationship between each connected domain based on the connected domain in the image to determine the starting sub-marker. In the marker, the frame, background and sub-marker form an enclosing relationship. If the sub-marker contains a feature point, the sub-marker and the feature point also have an enclosing relationship. Among them, the frame encloses the background, the background encloses the sub-marker, and the sub-marker encloses the feature point.
[0056] In some embodiments, the electronic device can identify each sub-marker in the marker by acquiring the connected domain information in the image. The connected domain refers to the image area composed of pixels with the same pixel value and adjacent positions in the image. The connected domain can be obtained by calculating the connected components marked as Boolean images using 4-way or 8-way connectivity, and the number of connected domains is output, wherein the type of each connected domain can be output according to the enclosing relationship, that is, the connected domain of each part such as the frame, background, sub-marker, and feature point in the output image. The connected domain corresponding to the frame can be defined as the first connected domain, the connected domain corresponding to the background can be defined as the second connected domain, the connected domain corresponding to the sub-marker can be defined as the third connected domain, and the connected domain corresponding to the feature point can be defined as the fourth connected domain. Among them, the connected domain that surrounds other connected domains and is not surrounded by any connected domain can be determined as the first connected domain; the connected domain surrounded by the first connected domain and adjacent to the first connected domain can be determined as the second connected domain; the connected domain surrounded by the second connected domain and adjacent to the second connected domain can be determined as the third connected domain; the connected domain surrounded by the third connected domain can be determined as the fourth connected domain. By identifying the enclosing relationship between each connected domain, the sub-marker information in the marker and the feature point information contained in each sub-marker can be obtained. As a specific implementation, a third connected domain containing two or more fourth connected domains can be determined as a starting sub-marker, and a third connected domain containing one or no fourth connected domain can be determined as a feature sub-marker. Therefore, by identifying the connected domain relationship of the marker image, the sub-marker information and the feature point information contained in each sub-marker can be directly identified.
[0057] In a specific embodiment, Figure 3 In the marker image shown, the frame is a connected domain, the background is a connected domain, the sub-marker is a connected domain, and each feature point in the sub-marker is a connected domain; the connected domain of the frame is recorded as X, the connected domain of the background is recorded as Y, the connected domain of the sub-marker is recorded as Z, and the connected domain of the feature point is recorded as O. Since the number of feature points of the starting sub-marker is different from the number of feature points of the feature sub-marker, the number of feature points of the feature sub-marker is 1 or 0, and the number of feature points of the starting sub-marker is 2, when the surrounding relationship of the connected domain in the image of the target marker is identified as XYZOO, the sub-marker corresponding to the Z connected domain can be determined as the starting sub-marker, where XYZOO means: X contains Y, Y contains Z, and Z contains two Os; when the surrounding relationship of the connected domain in the image of the target marker is identified as XYZO or XYZ, the sub-marker corresponding to the Z connected domain is not determined as the starting sub-marker, but the sub-marker corresponding to the Z connected domain is determined as the feature sub-marker.
[0058] In some embodiments, when determining the markers contained in the captured image, the markers present in the image can be determined based on the actual outer contour of the markers. Since there is a contrast between the color of the border and the background in the image, the target markers with the outer contour consistent with the markers can be obtained regardless of the intensity of the light in the captured environment background. For example, Figure 3 As shown, the frame of the marker is a square, the outer contour of the frame is also a square, the inner contour of the frame is a square, the outer contour of the background of the marker is also a square, and the frame of the marker is black and the background is white. When the ambient background is bright, the ambient background in the image is significantly different from the color or contrast of the frame, and the corresponding target marker can be obtained by identifying the outer contour of the frame. When the ambient background is dark, the ambient background in the image is slightly different from the color or contrast of the frame, but is significantly different from the color or contrast of the background, and the corresponding target marker can be obtained by identifying the outer contour of the background. Therefore, regardless of the light of the ambient background, the marker that may exist in the image can be determined based on the frame of the marker and the outer contour of the background. It should be noted that the shapes of the outer contour of the marker and the outer contour of the background are not limited, and can be designed according to the actual needs of the marker, for example, the outer contour of the marker is circular, and the outer contour of the background is square, etc. After determining the possible marker, the sub-markers in the marker are identified to obtain the sub-marker information contained in the marker, wherein the sub-marker information may include the number of sub-markers. For example, the background of the marker actually includes 8 sub-markers. In some embodiments, when the number of sub-markers contained in the detected background is greater than or equal to the number threshold, it can be determined as a target marker, and the starting sub-marker of the target marker is identified. For example, when the number of sub-markers detected is greater than or equal to 3, the starting sub-marker is identified. As an embodiment, the number threshold can be associated with the number of sub-markers actually contained in the target marker, and the number of sub-markers actually contained in the target marker can be pre-stored. The more sub-markers actually contained, the larger the number threshold can be, but it is not limited thereto.
[0059] Step S130: identifying a characteristic sub-marker of the target marker in the image, and acquiring second information of the characteristic sub-marker.
[0060] In some embodiments, the electronic device can identify each characteristic sub-marker contained in the marker based on the enclosing relationship of each connected domain in the marker, and obtain second information of each characteristic sub-marker, where the second information may include the number of feature points contained in each characteristic sub-marker, the pixel coordinates of each characteristic sub-marker in the image, etc.
[0061] Step S140: Determine the identity information of the target marker according to the first information and the second information.
[0062] The electronic device identifies the starting sub-marker and the characteristic sub-marker contained in the target marker, obtains the first information of the starting sub-marker and the second information of the characteristic sub-marker, and determines the relative position relationship between the starting sub-marker and each characteristic sub-marker in the image based on the first information and the second information, obtains the arrangement relationship of the characteristic sub-markers and the encoding information of each characteristic sub-marker, and then determines the identity information of the target marker.
[0063] In some embodiments, the pixel coordinates of the starting sub-mark and the pixel coordinates of each characteristic sub-mark may be obtained, and the relative positional relationship between the starting sub-mark and each characteristic sub-mark in the image may be determined according to the pixel coordinates of the starting sub-mark and the pixel coordinates of each characteristic sub-mark. Figure 3 For example, after obtaining the pixel coordinates of the starting sub-marker 231 and the pixel coordinates of each characteristic sub-marker 232 and 233, the relative position relationship between the starting sub-marker 231 and each characteristic sub-marker 232 and 233 in the image can be obtained. For example, the characteristic sub-marker 233 and a characteristic sub-marker 233 are arranged in the same row and are located on the right side of the starting sub-marker 231. A characteristic sub-marker 232 is arranged in the next row of the characteristic sub-marker 233, and so on. Therefore, the arrangement relationship of each characteristic sub-marker 232 and 233 can be obtained.
[0064] In some embodiments, the electronic device obtains the arrangement relationship of each characteristic sub-marker in the target marker, and can calculate the center of mass of each sub-marker, including the center of mass of each characteristic point of the starting sub-marker and the center of mass of the characteristic sub-marker, and establishes a direction vector between the center of mass of each characteristic sub-marker and the center of mass of the characteristic point of the starting sub-marker. The arrangement relationship of each characteristic sub-marker can be determined based on each direction vector. If the characteristic sub-marker contains a characteristic point, there is a center of mass corresponding to the characteristic point, which is the center of mass of the characteristic sub-marker. The coordinates of the center of mass corresponding to the sub-marker can be calculated based on the pixel coordinates of the characteristic point contained in the characteristic sub-marker in the image; if the characteristic sub-marker does not contain a characteristic point, there is a center of mass corresponding to the characteristic sub-marker. The coordinates of the center of mass corresponding to the sub-marker can be calculated based on the pixel coordinates of the characteristic sub-marker in the image. The specific method of calculating the center of mass is not limited in the embodiments of the present application. For example, it can be calculated according to a weight calculation method. Fig. 9 As described above, numbers 1 and 2 are the centroids of two characteristic points in the starting sub-marker, and numbers 3-9 are the centroids of each characteristic sub-marker.
[0065] In some embodiments, the arrangement relationship of each characteristic sub-marker in the target marker can be determined according to the following method. A characteristic point in the starting sub-marker is selected, and the centroid of the characteristic point is used as the starting point, and connected with any other centroid in the starting sub-marker to form a starting vector. The starting point of the starting sub-marker can be connected to the centroid of each characteristic sub-marker respectively to form a comparison vector between the centroid of each remaining characteristic sub-marker and the starting point, and the angle between all comparison vectors and the starting vector is obtained. If all angles are less than a preset first threshold, the arrangement relationship of the corresponding characteristic sub-markers is determined according to the size and modulus of each angle (the distance from the centroid of the characteristic sub-marker to the starting point). If there is an angle greater than the preset first threshold, the centroid of another characteristic point in the starting sub-marker is re-selected as the starting point, and the starting vector and comparison vector are constructed again to determine the arrangement relationship of the characteristic sub-markers. For example, Fig. 9 The partial vectors shown have a starting point of centroid 1, centroids 1 and 2 constitute a starting vector t1, centroids 1 and 3 constitute a comparison vector t2, centroids 1 and 5 constitute a comparison vector t3, and centroids 1 and 4 constitute a comparison vector t4, wherein the angle between the comparison vector t2 and the starting vector t1 is the minimum value of the angles between all comparison vectors and the starting vector t1, which is 0°, and the angle between the comparison vector t4 and the starting vector t1 is the maximum value of the angles between all comparison vectors and the starting vector t1, which is 90°; the preset first threshold is 91°, and all angles are less than the first threshold, so the arrangement relationship of the feature sub-markers can be calculated; the arrangement relationship can be arranged from small to large according to the angle value and the module length value from small to large, and the corresponding arrangement relationship of the feature sub-markers is 3695847.
[0066] In some implementations, the centroid of a starting sub-marker can be calculated based on the two feature points of the starting sub-marker, and a direction vector between the centroid of each feature sub-marker and the centroid of the starting sub-marker can be established. The arrangement relationship of each feature sub-marker can be determined based on each direction vector. Specifically, the centroid of the starting sub-marker is used as the starting point, and the lines connecting the centroids of all feature sub-markers and the starting point form a vector, and the modulus of all vectors is obtained. The arrangement relationship of the corresponding feature sub-markers is determined based on the size of the modulus. For example, Fig.10 The partial vectors shown are arranged from small to large according to the value of the modulus, and the corresponding arrangement relationship of the feature sub-markers is 4536789.
[0067] In some embodiments, after obtaining the arrangement relationship of each characteristic sub-marker in the target marker, the starting sub-marker can be used as a reference point, and the codes corresponding to each characteristic sub-marker can be obtained in sequence according to the arrangement relationship in a specific order to obtain a code string formed by each characteristic sub-marker in a certain order. As a specific embodiment, the characteristic sub-marker can correspond to a binary code, wherein the code of the characteristic sub-marker containing 1 characteristic point is "1", and the code of the characteristic sub-marker not containing a characteristic point is "0". It can be understood that the encoding method can be various, and is not limited to this.
[0068] by Figure 3 and Figure 4A For example, the binary code of the characteristic sub-marker 232 is "1", and the binary code of the characteristic sub-marker 233 is "0". Figure 4A As shown, taking the starting sub-marker as the reference point, the codes of the characteristic sub-markers are obtained in sequence from 1 to 7 in order, and the code string of the characteristic sub-marker of the marker in FIG4 C can be obtained as "0110001". Figure 4D The encoding string of the characteristic sub-tag of the marker is "0110110".
[0069] In some embodiments, the electronic device may use the code string of the characteristic sub-marker as the code string of the target marker. The code string of the target marker may also be a combination of the code string of the starting sub-marker and the code string of the characteristic sub-marker. The form of the code string of the target marker is not limited here. The code of the starting sub-marker may be associated with the number of characteristic points contained in the starting sub-marker. For example, the code of the starting sub-marker containing 2 characteristic points is 2, and the code of the starting sub-marker containing 3 characteristic points is 3, etc. Figure 4C For example, the coding string of the marker may be a combination of the coding of the characteristic sub-marker and the coding string of the characteristic sub-marker, which may be "20110001". It is understandable that the coding method of the target marker may be other methods, and the coding of the start sub-marker and the characteristic sub-marker is not limited to the above methods.
[0070] In some embodiments, the electronic device may obtain the identity information of the target marker based on the pre-stored coding information and the acquired coding string of the target marker. The coding information of each marker is pre-stored in the memory of the electronic device, and the electronic device may compare the coding string of the target marker in the image with the pre-stored coding information, and determine the pre-stored marker corresponding to the marker. When the coding information of the pre-stored marker is consistent with the coding string of the target marker, it can be determined that the consistent pre-stored marker matches the target marker. When there is no pre-stored marker whose coding information is consistent with the coding string of the target marker, the similarity between the coding string of the target marker and the coding information of each pre-stored marker can be calculated, and the pre-stored marker with the highest similarity can be selected as the pre-stored marker that matches the target marker. There are many ways to calculate similarity. For example, the Hamming distance between two codes can be calculated, and the codes at the same sequence position can be compared one by one. The Hamming distance is calculated according to the number of different codes. For example, the code string of the target marker is "0110001" and the code information of the pre-stored marker is "0110011". The two codes are compared from the first bit. If the codes at the sixth bit are different, the Hamming distance of the two codes is 1. The method for calculating similarity can also be other methods, and is not limited to the above method.
[0071] In one embodiment, if there is only one marker with a starting sub-marker having a specific number of characteristic points in the pre-stored markers, then when the electronic device identifies that the target marker contains the starting sub-marker with the specific number of characteristic points, the identity information of the target marker can be determined without identifying other sub-markers. In some embodiments, after the electronic device obtains a pre-stored marker that matches the target marker, the identity information of the matching pre-stored marker can be used as the identity information of the target marker, and the identity information may include coding information, physical coordinates of the starting sub-marker and the characteristic sub-marker in the marker, physical coordinates of each characteristic point, etc., wherein the physical coordinates refer to the coordinates in the physical coordinate system corresponding to the target marker, and the physical coordinates of the starting sub-marker and the characteristic sub-marker can be used to represent the actual position of the starting sub-marker and the characteristic sub-marker in the target marker.
[0072] In some embodiments, the position and posture information of the target marker relative to the electronic device can be obtained according to the pixel coordinates of the starting sub-marker and the characteristic sub-marker in the image, and the physical coordinates of the starting sub-marker and the characteristic sub-marker. As a specific embodiment, the electronic device can obtain the mapping parameters between the image coordinate system and the physical coordinate system according to the pixel coordinates, physical coordinates and the internal parameters of the image sensor acquired in advance of each sub-marker, wherein the image coordinate system can be the pixel plane coordinate system of the image, and the pixel coordinates can be used to represent the position of the point in the image. According to the mapping parameters between the image coordinate system and the physical coordinate system, the rotation parameters and translation parameters between the camera coordinate system of the image sensor and the physical coordinate system can be obtained, and the rotation parameters and translation parameters are used as the position and posture information between the image sensor and the target marker. The rotation parameters and translation parameters are the six-degree-of-freedom information of the image sensor in the physical coordinate system, which can represent the rotation and movement state of the image sensor in the physical coordinate system, so that the position and posture information of the target marker relative to the electronic device can also be obtained.
[0073] The marker provided by the present application can be identified faster and more accurately, and is suitable for automatic and batch generation, which solves the use limitations of existing markers, and can adjust the size of the marker according to the needs. As the size of the marker changes, the arranged sub-markers can be automatically increased or decreased. For example, in a 3X3 size marker, 3 sub-markers are arranged in each row, and in a 5X5 size marker, 4 or 5 sub-markers can be arranged in each row. The marker of the present application has specific rules, the position of the characteristic sub-marker in the marker is relatively fixed, and the difference between markers with different identity codes lies in whether there is a difference in the characteristic points in the characteristic sub-marker. As long as there is a difference in the characteristic sub-marker, it can be distinguished. In the process of automatic generation, only the characteristic sub-marker information needs to be changed, which is suitable for batch production; the marker structure of the present application can set a large number of different identity codes, which is far more than the number of different identity codes that can be set for other markers; when the demand for the number of markers with different identity codes is not large, the similarity between the generated markers must be greater than a certain value to prevent interference caused by approximation.
[0074] See also Fig.11, which shows a structural block diagram of a recognition device 500 for identifying a marker provided by an embodiment of the present application, which is applied to a terminal device, and the device may include: an acquisition module 510, a first recognition module 520, a second recognition module 530, and a determination module 540. Among them, the acquisition module 510 is used to acquire an image containing a target marker; the first recognition module 520 is used to identify the starting sub-tag of the target marker in the image and obtain the first information of the starting sub-tag; the second recognition module 530 is used to identify the characteristic sub-tag of the target marker in the image and obtain the second information of the characteristic sub-tag; the determination module 540 is used to determine the identity information of the target marker according to the first information and the second information.
[0075] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.
[0076] In several embodiments provided in the present application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or modules, which may be electrical, mechanical or other forms.
[0077] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.
[0078] Please refer to Fig.12 , which shows a structural block diagram of a terminal device provided in an embodiment of the present application. The terminal device 100 may be a terminal device such as a smart phone, a tablet computer, a head-mounted display device, etc. that can run applications. The terminal device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, an image sensor 130, and one or more applications, wherein one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and one or more programs are configured to execute the method described in the aforementioned method embodiment.
[0079] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire terminal device 100, and executes various functions and processes data of the terminal device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.
[0080] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the terminal device 100 during use, etc.
[0081] In the embodiment of the present application, the image sensor 130 is used to capture images of physical objects and scene images of target scenes. The image sensor 130 can be an infrared camera or a color camera, and the specific camera type is not limited in the embodiment of the present application.
[0082] A block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the method described in the above method embodiment. The computer-readable storage medium can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program codes that execute any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code can be compressed, for example, in an appropriate form.
[0083] The present application also provides an interactive device, such as Fig.13 As shown, the interactive device 30 is held by a user 40, and the interactive device 30 includes a main body 31 and a tracking unit 32. The user 40 can hold the main body 31 and interact with the virtual content by manipulating the main body 31. The main body 31 can be provided with a control structure with a control function such as a button, a trigger, a touch pad, etc., which is not specifically summarized in this application; the tracking unit 32 is connected to the main body 31, and the positions of the tracking unit 32 and the main body 31 are relatively fixed. The tracking unit 32 can be connected to the main body 31 in a detachable or integrally formed manner. A plurality of markers 33 are provided on the tracking unit 32, and the plurality of markers 33 are dispersedly arranged on the outer surface of the tracking unit 32.
[0084] See also Fig.14 , is a schematic diagram of a tracking marker of the present application, wherein the terminal device 100 is provided with an image sensor, and the marker 33 is located within the visual range of the image sensor of the terminal device 100. In a specific embodiment, the image sensor is an infrared camera, which collects infrared light; a light source 34 is provided in the tracking portion 32 of the interactive device 30, and the light source 34 is an infrared light source; the marker 33 has a base layer 331 and an identification layer 332, and the base layer 331 and the identification layer 332 form a mark. Among them, the base layer 331 is made of a hard infrared-transmitting material and is in the shape of a plate, so that infrared light can pass through the base layer 331, but visible light cannot pass through the base layer 331; the identification layer 332 is made of an opaque material, which is adhered to the base layer 331, and the thickness of the identification layer 332 is less than the thickness of the base layer 331. Figure 3In the figure, the frame 21 and the sub-marker 23 of the marker 20 are the identification layer 332, and the background 22 and the feature points 2311 and 2321 are directly formed by the combination of the base layer 331 and the identification layer 332. When the light source 34 emits light, the light that passes through the base layer 331 and is not blocked by the identification layer 332 is collected by the terminal device 100, and an image of the marker can be formed.
[0085] In some embodiments, Figure 3 The background and feature points in the image are set as the identification layer 332, and the border and the sub-mark are directly formed by combining the base layer 331 and the identification layer 332.
[0086] In some embodiments, the light source 34 may not be an infrared light source, but may be other invisible light sources, such as ultraviolet light. The image sensor of the terminal device 100 needs to be correspondingly configured as a camera for collecting ultraviolet light. The light source 34 may also be a visible light source, and the image sensor of the terminal device 100 needs to be correspondingly configured as a camera for collecting visible light.
[0087] In some embodiments, Fig.15 As shown, a light guide 35 may be provided between the light source 34 and the marker 33 so that the light is evenly directed toward the marker 33, and finally a clearly visible marker image is formed.
[0088] In some embodiments, Fig.16 As shown, the interactive device is provided with a plurality of non-coplanar markers 33, a single light source 34 and a light guide 36. The light guide 36 has a light guide portion (not marked) corresponding to each marker 33, each light guide portion gradually shrinks from the marker 33 toward the light source 34, and adjacent light guide portions are connected at one end close to the light source 34. After the light of a single light source 34 is guided by each light guide portion, it is evenly scattered to each corresponding marker 33.
[0089] It should be noted that the form of the interactive device is not limited to Fig.11 The interactive device can be in any form. The specific form design can be designed according to the actual content scene, and the position arrangement of the markers can also be designed according to the actual situation.
[0090] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not drive the essence of the corresponding technical solutions away from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for identifying a marker, characterized in that: The method comprises: Acquire an image containing a target marker, wherein the target marker includes a background and a plurality of sub-markers distributed on the background, wherein the plurality of sub-markers include a starting sub-marker and one or more characteristic sub-markers, wherein the position of the characteristic sub-marker relative to the starting sub-marker is fixed, and the characteristic sub-marker has corresponding coding information; Identify a starting sub-marker of a target marker in the image, and obtain first information of the starting sub-marker, wherein the starting sub-marker includes at least two feature points, and the first information includes pixel coordinates of the starting sub-marker; Identify a characteristic sub-marker of a target marker in the image, and obtain second information of the characteristic sub-marker, wherein the characteristic sub-marker includes a characteristic point or does not include a characteristic point, and the second information includes pixel coordinates of each characteristic sub-marker; Determine the relative position relationship between the starting sub-marker and each characteristic sub-marker in the image according to the first information and the second information; Acquire the arrangement relationship between the characteristic sub-markers according to the relative position relationship; Obtaining encoding information corresponding to each characteristic sub-tag; The identity information of the target marker is determined according to the arrangement relationship between the characteristic sub-markers and the coding information corresponding to each characteristic sub-marker.
2. The method according to claim 1, characterized in that The step of identifying a starting sub-marker of a target marker in the image comprises: Acquire each connected domain in the image; The enclosing relationship between the connected domains is determined, and the starting sub-tag is identified according to the enclosing relationship, and the sub-tag corresponding to the connected domain that is enclosed by the connected domain corresponding to the background and encloses at least two other connected domains is used as the starting sub-tag.
3. The method according to claim 1, characterized in that The identifying characteristic sub-markers of the target marker in the image includes: Acquire each connected domain in the image; The enclosing relationship between the connected domains is determined, and the characteristic sub-marker is identified according to the enclosing relationship, and the sub-marker corresponding to the connected domain that is enclosed by the background and that encloses another connected domain or that does not enclose other connected domains is taken as the characteristic sub-marker.
4. The method according to claim 1, characterized in that: The determining the relative position relationship between the starting sub-marker and each characteristic sub-marker in the image according to the first information and the second information includes: Calculating the centroid of each feature point in the starting sub-marker according to the pixel coordinates of the starting sub-marker, and calculating the centroid of the corresponding feature sub-marker according to the pixel coordinates of each feature sub-marker; Establishing a direction vector between the centroid of each characteristic sub-marker and the centroid of the characteristic point of the starting sub-marker respectively; The relative position relationship between the starting sub-marker and each characteristic sub-marker in the image is determined according to each established direction vector.
5. The method according to claim 1, characterized in that The determining the identity information of the target marker according to the arrangement relationship between the characteristic sub-markers and the coding information corresponding to the characteristic sub-markers includes: Taking the starting sub-mark as a reference point, sequentially acquiring the coding information corresponding to each characteristic sub-mark according to the arrangement relationship in a predetermined order; forming a code string according to the code information obtained in the predetermined order; The coding string is compared with the coding information of the pre-stored marker to determine the identity information of the target marker.
6. The method according to claim 5, characterized in that The step of comparing the coding string with the pre-stored coding information of the marker to determine the identity information of the target marker includes: Calculating the similarity between the encoding string and the encoding information of each pre-stored marker; The pre-stored marker with the highest similarity is selected as the pre-stored marker matching the target marker, and the identity information of the matching pre-stored marker is used as the identity information of the target marker.
7. A marker, used for image acquisition by an image sensor for identification and tracking, characterized in that: The marker includes a background and multiple sub-markers distributed in the background; the multiple sub-markers include a starting sub-marker and one or more characteristic sub-markers, the starting sub-marker includes at least two characteristic points, each of the characteristic sub-markers includes one characteristic point or no characteristic point, the position of the characteristic sub-marker relative to the starting sub-marker is fixed, and the characteristic sub-marker has corresponding coding information.
8. The marker according to claim 7, characterized in that The color of the background is different from the color of the sub-mark, and the color of the sub-mark is different from the color of the feature point.
9. A marker identification device, characterized in that: include: an acquisition module, configured to acquire an image containing a target marker, wherein the target marker includes a background and a plurality of sub-markers distributed on the background, wherein the plurality of sub-markers include a starting sub-marker and one or more characteristic sub-markers, wherein the position of the characteristic sub-marker relative to the starting sub-marker is fixed, and the characteristic sub-marker has corresponding coding information; A first recognition module is used to recognize a starting sub-marker of a target marker in the image and obtain first information of the starting sub-marker, wherein the starting sub-marker includes at least two feature points, and the first information includes pixel coordinates of the starting sub-marker; A second recognition module is used to recognize a characteristic sub-marker of the target marker in the image and obtain second information of the characteristic sub-marker, wherein the characteristic sub-marker includes one characteristic point or does not include a characteristic point, and the second information includes pixel coordinates of each characteristic sub-marker; A determination module is used to determine the relative position relationship between the starting sub-marker and each characteristic sub-marker in the image according to the first information and the second information; obtain the arrangement relationship between the each characteristic sub-marker according to the relative position relationship; obtain the coding information corresponding to the each characteristic sub-marker; and determine the identity information of the target marker according to the arrangement relationship between the each characteristic sub-marker and the coding information corresponding to the each characteristic sub-marker.
10. A terminal device, characterized in that: include: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1-6.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Image processing method and device, and electronic equipment
CN103634587A
Dynamic registration method for enhanced real field spaces and virtual three-dimensional targets
CN106249881A
Method, device and system for searching stripe set
CN107111882A
FPGA-based infrared salient object detection method
CN107194946A
Image identification method and terminal
CN107665324A
