Pattern mapping methods of multi-curvature pattern marker

Abstract

One embodiment of the present invention provides a pattern marker mapping method comprising the steps of: receiving a plurality of image frames captured at different times from a capturing device for capturing a pattern marker; detecting a pattern portion of a pattern marker from each of the plurality of image frames; and mapping the detected pattern portions to a matching coordinate system. The step of detecting the pattern portion involves detecting a reference sequence portion representing first information and a unique sequence portion representing second information different from the first information from the pattern portion. The step of mapping to a matching coordinate system involves mapping the detected reference sequence portion to an index value on the matching coordinate system corresponding to the second information of the detected unique sequence portion.

Description

Technology Field

[0001] The present invention relates to a pattern mapping method for multi-curvature pattern markers. Background Technology

[0002] Generally, in motion tracking systems, the position of an object was measured by attaching an identification device called a marker to the object and tracking the marker. With the advancement of image processing technology, moving beyond the conventional method of simply utilizing images acquired through a single camera, methods that produce three-dimensional stereo images by photographing an object with multiple cameras and aligning the captured images are widely used in recent years.

[0003] Conventionally, in the process of acquiring marker points from an image of a three-dimensional pattern marker and mapping them to a virtual coordinate system, there was a problem in that a large error occurred between the position value of the marker point in the absolute coordinate system and the position value of the marker point mapped to the virtual coordinate system due to the influence of various environmental changes, such as changes in lighting in the shooting environment and irregular changes in the user's posture.

[0004] To resolve these issues, research and development have been ongoing to reduce the error value by equipping multiple imaging devices and simultaneously capturing marker points from different angles; however, this approach has drawbacks, such as the requirement for a large number of expensive cameras and the occurrence of errors during the calibration process from the coordinate systems of each of the multiple imaging devices to a single matching coordinate system. The problem to be solved

[0005] The technical problem to be solved by the present invention is to provide a pattern mapping method in which a first element is mapped to an accurate index value of a matching coordinate system by an information processing device, by fusing position information of a first element obtained by a shooting device capturing an area of ​​a pattern part at a preset time and position information of a first element obtained by capturing the area of ​​the pattern part at a time different from the preset time. means of solving the problem

[0006] One aspect of the present invention provides a pattern marker mapping method comprising the steps of: receiving a plurality of image frames captured at different times from a capturing device for capturing a pattern marker; detecting a pattern portion of a pattern marker from each of the plurality of image frames; and mapping the detected pattern portions to a matching coordinate system. The step of detecting the pattern portions includes detecting a reference sequence portion representing first information and a unique sequence portion representing second information different from the first information from the pattern portions, and the step of mapping to a matching coordinate system includes mapping the detected reference sequence portion to an index value on the matching coordinate system corresponding to the second information of the detected unique sequence portion.

[0007] Additionally, the reference sequence section has a plurality of first elements, and the unique sequence section has a plurality of second elements, and the reference sequence section and the unique sequence section are arranged sequentially along a first direction of the pattern marker, and each of the plurality of first elements and the plurality of second elements is arranged along a second direction different from the first direction, and the arrangement of the plurality of second elements may follow a de bruijn sequence.

[0008] Additionally, the pattern marker is formed in a cylindrical shape, the first direction is parallel to the longitudinal central axis of the pattern marker, and the second direction may be the circumferential direction of the pattern marker.

[0009] Additionally, the step of receiving a plurality of image frames includes the step of receiving a first image frame in which an area of ​​a pattern marker is captured at a first time point, and the step of receiving a second image frame in which an area of ​​a pattern marker is captured at a second time point different from the first time point. The step of detecting a pattern portion may include the step of detecting a first element included in the first image frame, the step of detecting an array of second elements corresponding to the first element included in the first image frame, the step of detecting a first element included in the second image frame, and the step of detecting an array of second elements corresponding to the step of detecting the first element included in the second image frame.

[0010] Additionally, the step of detecting a first element included in a first image frame includes the step of detecting a first index value in a matching coordinate system using an array of second elements corresponding to the first element included in the first image frame, and the step of detecting a first element included in a second image frame may include the step of detecting a second index value in a matching coordinate system using an array of second elements corresponding to the first element included in the second image frame.

[0011] In addition, the first index value and the second index value may follow a preset probability distribution.

[0012] In addition, the pre-set probability distribution can be a Gaussian distribution.

[0013] Additionally, the step of mapping to a matching coordinate system may include the step of calculating the weight of a first index value and the weight of a second index value using a preset probability distribution, and the step of calculating the index value to which a first element included in one region of a pattern marker is mapped onto the matching coordinate system using the first index value, the weight of the first index value, the second index value, the weight of the second index value, and a preset algorithm.

[0014] In addition, the preset algorithm can be a linear Kalman filter.

[0015] Another aspect of the present invention provides a pattern marker mapping system comprising a pattern marker of a three-dimensional shape having a pattern formed around its perimeter, a shooting device for shooting the pattern marker, and an information processing device, wherein the information processing device performs the steps of receiving a plurality of image frames shot at different times from the shooting device for shooting the pattern marker, detecting a pattern portion of the pattern marker from each of the plurality of image frames, and mapping the detected pattern portions to a matching coordinate system. Effects of the invention

[0016] An information processing device according to one embodiment of the present invention can calculate index values ​​on a matching coordinate system of a pattern part from image frames captured at different times and fuse the calculated index values ​​according to a preset algorithm. As a result, the information processing device can derive highly reliable mapping index values, thereby enabling the pattern part to be mapped to highly reliable index values ​​on the matching coordinate system. Brief explanation of the drawing

[0017] FIG. 1 is a drawing illustrating a pattern marker attached to a body part of an object according to one embodiment of the present invention. FIG. 2 is a perspective view of a pattern marker according to one embodiment of the present invention. FIG. 3 is a drawing for explaining an example configuration of a pattern part according to one embodiment of the present invention. FIG. 4 is a drawing for explaining the first to third elements according to an embodiment of the present invention. FIGS. 5 and 6 are drawings for explaining a shooting area according to an embodiment of the present invention. FIG. 7 is a diagram illustrating a method for mapping pattern parts to a matching coordinate system according to an embodiment of the present invention. FIG. 8 is a drawing illustrating a cylindrical pattern marker located on an absolute coordinate system according to one embodiment of the present invention. FIG. 9 is a flowchart illustrating a method for an information processing device according to an embodiment of the present invention to receive a plurality of image frames and map pattern parts to a matching coordinate system. Specific details for implementing the invention

[0018] The present invention is capable of various modifications and may have various embodiments; specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described below in detail together with the drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various forms.

[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0020] In the following examples, singular expressions include plural expressions unless the context clearly indicates otherwise.

[0021] In the following embodiments, terms such as "include" or "have" mean that the features or components described in the specification are present, and do not preclude the possibility that one or more other features or components may be added.

[0022] Where an embodiment can be implemented differently, a specific process sequence may be performed differently from the order described. For example, two processes described consecutively may be performed substantially simultaneously or proceed in the reverse order of the description.

[0023] In the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are depicted arbitrarily for convenience of explanation, so the following embodiments are not necessarily limited to those illustrated.

[0024] FIG. 1 is a drawing illustrating a pattern marker attached to a body part of an object according to an embodiment of the present invention, FIG. 2 is a perspective view of a pattern marker according to an embodiment of the present invention, and FIG. 3 is a drawing for explaining an exemplary configuration of a pattern part according to an embodiment of the present invention. FIG. 4 is a drawing for explaining first to third elements according to an embodiment of the present invention, FIG. 5 and FIG. 6 are drawings for explaining a shooting area according to an embodiment of the present invention, and FIG. 7 is a drawing for explaining a method of mapping pattern parts to a matching coordinate system according to an embodiment of the present invention. FIG. 8 is a drawing illustrating a cylindrical pattern marker located on an absolute coordinate system according to an embodiment of the present invention, and FIG. 9 is a flowchart for explaining a method in which an information processing device according to an embodiment of the present invention receives a plurality of image frames and maps pattern parts to a matching coordinate system.

[0025] Referring to FIG. 1, a pattern marker (M) according to one embodiment of the present invention may be attached to a preset area of ​​an object to be motion captured, and the position and posture of the pattern marker (M) may be tracked by a separately configured shooting device (C).

[0026] A pattern portion (100) containing information regarding the position and orientation of the pattern marker (M) may be formed on one side of the pattern marker (M), and an information processing device (P) may recognize the pattern portion (100) of the pattern marker (M) from an image obtained by a shooting device (C) and calculate the position and orientation of the pattern marker (M). As a result, the information processing device (P) can calculate the position and orientation of the pattern marker (M) and predict the position and orientation of an object to which the pattern marker (M) is attached or which is located adjacent to the pattern marker (M).

[0027] The shape and configuration of the pattern part (100) and the method for precisely calculating information on the position and orientation of the pattern marker (M) from the pattern part (100), which enables the information processing device (P) to recognize the pattern part (100) and precisely calculate the position and orientation of the pattern marker (M), are described in detail below in the specification.

[0028] A pattern marker (M) according to one embodiment of the present invention may be an active marker that emits light on its own, and when the pattern markers (M) are used as active markers, the robot does not need to use a separate light source. Alternatively, the pattern marker (M) may be a passive marker that reflects light emitted from at least one light source. In this case, the pattern marker (M) may reflect light emitted from an external light source. In one embodiment, depending on the purpose of the information processing device (P) tracking human body motion information and the environment in which motion capture is performed, the pattern marker (M) may be used as a combination of active markers and passive markers.

[0029] According to another embodiment, the pattern marker (M) may be composed of a sensor that is the target of magnetic motion capture. In this case, a magnetic field generating device may be separately provided in the space where motion capture is performed, and the position and orientation of the pattern marker (M) can be calculated by calculating the amount of change in the magnetic field according to the movement of the pattern marker (M).

[0030] According to another embodiment, the pattern marker (M) may be composed of an inertial sensor that is the subject of inertial motion capture. In this case, the information processing device (P) can calculate the position and attitude according to the movement of the pattern marker (M).

[0031] The pattern marker (M) can be a 2D marker as well as a 3D marker, and specifically, the surface of the pattern marker (M) that emits or reflects light can be formed in 2D or 3D.

[0032] Multiple pattern markers (M) may be attached to the human body that is the subject of motion capture. Each of the multiple pattern markers (M) may have a different pattern formed on it, and the pattern markers (M) may emit or reflect light in correspondence with the shape of the pattern.

[0033] Information regarding the pattern portion (100) formed on a plurality of pattern markers (M) can be stored in a separately provided information processing device (P), and the information processing device (P) can assign a marker number corresponding to the information regarding the pattern portion (100) to each pattern marker (M). Accordingly, the information processing device (P) can receive light corresponding to the shape of the pattern portion (100), extract the corresponding marker number, and identify a plurality of pattern markers (M).

[0034] The pattern marker (M) can form a unique coordinate system. In one embodiment, the pattern marker (M) can form a virtual three-dimensional grid coordinate system consisting of x-axis, y-axis, and z-axis that are orthogonal to each other, with a point pre-set on the pattern marker (M) as the origin. In this case, the information processing device (P) can measure geometric information of the pattern marker (M), such as the position value and rotation value of the pattern marker (M), by using information on the relative geometric relationship between the x-axis, y-axis, and z-axis of the three-dimensional grid coordinate system formed by the pattern marker (M) and a shooting device (C) that is provided separately.

[0035] The pattern marker (M) may be included in a wearable device, and the wearable device may include at least one of an accessory type (e.g., a watch, ring, bracelet, anklet, necklace, glasses, contact lens, or head-mounted device (HMD)), a fabric or clothing integrated type (e.g., electronic clothing), a body-attached type (e.g., a skin pad or tattoo), or a bio-implantable circuit.

[0036] An information processing device (P) according to one embodiment of the present invention may be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and other electrical units for performing functions.

[0037] The information processing device (P) can create a virtual space having three-dimensional grid coordinates. The three-dimensional grid coordinates may be coordinates having mutually orthogonal x-axis, y-axis, and z-axis, but are not limited thereto, and include all grid coordinates capable of representing coordinates in three-dimensional space. The information processing device (P) can set a three-dimensional grid coordinate system in the virtual space by receiving information regarding the position and shooting direction of the provided shooting device (C).

[0038] Specifically, the information processing device (P) can generate a matching coordinate system as a virtual coordinate system that maps the coordinate system of the imaging device (C) and the pattern part (100) obtained from the imaging device (C) as a three-dimensional spatial coordinate system with the imaging device (C) as the origin. In addition, the information processing device (P) can calibrate the coordinate values ​​input into the coordinate system of the imaging device (C) onto the matching coordinate system.

[0039] A main body according to one embodiment of the present invention forms the exterior of a pattern marker (M) and may be made of a metal or non-metal material and may be formed integrally with a wearable device.

[0040] Referring to FIG. 3, a main body according to one embodiment of the present invention may include a pattern part (100) having a reference sequence part (150) and an encoding sequence part (110), and the pattern part (100) may be disposed on the surface of the main body.

[0041] The pattern section (100) may include a reference sequence section (150) and a unique sequence section (110) arranged sequentially along a first direction. In one embodiment, the first direction may be set to the y-axis direction (see FIG. 3), but is not limited thereto, and the first direction may be set to various directions such that the reference sequence section (150) and the unique sequence section (110) can be arranged in parallel.

[0042] The pattern section (100) may further include a distinguishing sequence section (170) located between the reference sequence section (150) and the unique sequence section (110). Specifically, the unique sequence section (110), the distinguishing sequence section (170), and the reference sequence section (150) may be arranged sequentially along a first direction.

[0043] As a result, the distinction sequence (170) section is positioned between the unique sequence section (110) and the reference sequence section (150), so that the area where the unique sequence section (110) is located and the area where the reference sequence section (150) is located can be spaced apart by a predetermined interval. Through this, errors can be prevented in the process of the imaging device (C) or information processing device (P) extracting the first information contained in the reference sequence section (150) and the second information contained in the unique sequence section (110) due to light interference effects between the unique sequence section (110) and the reference sequence section (150).

[0044] Referring to FIG. 3, a reference sequence unit (150) according to one embodiment of the present invention may be composed of a plurality of first elements (E1) representing first information.

[0045] A plurality of first elements (E1) may be arranged along a second direction, and the second direction may be set differently from the first direction. In one embodiment, the second direction may be set perpendicular to the first direction, and if the first direction is the y-axis direction (see FIG. 3), the second direction may be set to the x-axis direction (see FIG. 3).

[0046] Referring to FIG. 8 as another embodiment, the first direction may be set parallel to the first reference axis (AX1), in which case the second direction may be the circumferential direction of the main body.

[0047] The first information may include orientation information of the main body or the pattern marker (M). Specifically, if the main body is formed in a cylindrical shape with a length extended in the first direction, the first information may include orientation information including curvature information of the surface of the cylindrical main body. The description regarding the orientation information will be explained in detail in the description regarding the cylindrical main body including the first element (E1) and the second element (E2) to be described later.

[0048] A first element (E1) according to one embodiment of the present invention may include a first-1 element (E1-1) and a first-2 element (E1-2) made of materials having different characteristics.

[0049] In one embodiment, when the pattern marker (M) is composed of a passive marker, the first-1 element (E1-1) and the first-2 element (E1-2) may be made of materials with different reflectances, and in this case, the first-2 element (E1-2) may be made of a material with a relatively higher light reflectance than the first-1 element (E1-1).

[0050] In one embodiment, when the pattern marker (M) is composed of an active marker, the first-1 element (E1-1) may be made of a material that does not emit light, and the first-2 element (E1-2) may be made of a light source capable of emitting light.

[0051] As a result, since the imaging device (C) can detect light from the first-2 element (E1-2) but cannot detect light from the first-1 element (E1-1), the imaging device (C) or the information processing device (P) can detect the arrangement of the first element (E1) using the light obtained from the first-2 element (E1-2).

[0052] Referring to FIG. 4, when the imaging device (C) detects light from elements (E1, E2), it recognizes the elements (E1, E2) where light is detected as "1", and if it does not detect light, it recognizes them as "0". In one embodiment, the information processing device (P) can recognize the first-2 element (E1-2) as "1" and the first-1 element (E1-1) as "0".

[0053] Below is a technical description regarding the arrangement of the first element (E1) of the reference sequence section (150).

[0054] Referring to FIG. 3, the reference sequence section (150) may consist of a plurality of reference sequence sections (150) arranged along a first direction. In one embodiment, the reference sequence section (150) may consist of a first reference sequence section (150a) and a second reference sequence section (150b).

[0055] Hereinafter, the reference sequence section (150) is defined as a collective term for the first and second reference sequence sections (150).

[0056] In one embodiment, the reference sequence section (150) may be composed of an array of multiple first-second elements (E1-2) spaced apart at predetermined intervals along a second direction.

[0057] As a result, when the main body is formed in a cylindrical shape with a second direction in the circumferential direction, the reference sequence part (150) is formed in an arrangement of first and second elements (E1-2) capable of reflecting or emitting light, thereby having the effect that the imaging device (C) can detect the reference sequence part (150) from the pattern marker (M) regardless of the position and rotation angle of the pattern marker (M).

[0058] In one embodiment, the reference sequence section (150) may be composed of an array of repeating first-1 elements (E1-1) and first-2 elements (E1-2) spaced apart at predetermined intervals along a second direction. Specifically, the first-2 element (E1-2) is located in the area adjacent to the second direction of the first-1 element (E1-1), and the first-1 element (E1-1) is located in the area adjacent to the second direction of the first-2 element (E1-2), so that the array may be formed as an array that repeats at intervals of two first elements (E1).

[0059] However, it is not limited to this, and the repeating unit can be varied in various ways, such as the arrangement of the reference sequence section (150) being repeated at intervals of three first elements (E1), depending on the size of the main body, the size of the first element (E1), and the spacing between each first element (E1).

[0060] In one embodiment, a plurality of first elements (E1) may be composed of an array of binary codes. Specifically, a plurality of first elements (E1) may be composed of an array of Nth-order binary codes, where N can be one of 0, 1, or 2.

[0061] Specifically, when multiple first elements (E1) are arranged in an array of first binary codes, the reference sequence section (150) may be arranged in an array of repeating first-1 elements (E1-1) and first-2 elements (E1-2).

[0062] In this case, the information processing device (P) may recognize the reference sequence section (150) as an array of “0101010…”, and when a plurality of first elements (E1) are arranged in an array of second binary codes, the reference sequence section (150) may be composed of two first-1 elements (E1-1) and two first-2 elements (E1-2) arranged sequentially, so that the information processing device (P) may recognize the reference sequence section (150) as an array of “001100110011…”.

[0063] In one embodiment, the first reference sequence section (150a) may be formed in an array that is repeated at intervals of two first elements (E1), such that a first-2 element (E1-2) is located in an area adjacent to the second direction of the first-1 element (E1-1), and a first-1 element (E1-1) is located in an area adjacent to the second direction of the first-2 element (E1-2).

[0064] In one embodiment, the reference sequence section (150) may be composed of a combination of multiple reference sequence sections (150) each having a different arrangement. Additionally, the second reference sequence section (150b) may be composed of an arrangement consisting only of multiple first-second elements (E1-2) spaced apart at a predetermined interval.

[0065] As a result, the shooting device (C) or the information processing device (P) can calculate the shooting area (AR) by combining the reference sequence sections (150) of different arrangements, and additionally, the information processing device (P) can recognize the positions of the first elements (E1) by detecting light from the first elements (E1), specifically the first-second elements (E1-2), included in the shooting area (AR), so that the position values ​​of the first-second elements (E1-2) can be mapped to a matching coordinate system using the light detected from the first-second elements (E1-2).

[0066] The process of an information processing device (P) calculating an image area (AR), the process of generating a coordinate system of a pattern marker (M) using the image area (AR), and the method of tracking the attitude of a pattern marker (M) are described in detail below.

[0067] In one embodiment, the information processing device (P) can detect the location of the reference sequence section (150) and the arrangement of the first element (E1) from all detected elements (E1, E2), and can detect a unique sequence section (110) corresponding to the first direction using the detected reference sequence section (150) and the arrangement of the first element (E1).

[0068] Specifically, the information processing device (P) receives multiple image frames (IM) captured at different times from the shooting device (C) and can detect first elements (E1) included in each of the multiple image frames (IM), and can detect an array of second elements (E2) corresponding in a first direction to the first elements (E1) detected from each of the image frames (IM).

[0069] In one embodiment, depending on the geometric positional relationship between the imaging device (C) and the cylindrical pattern marker (M), the number of imaging areas (AR) that the information processing device (P) can detect may be multiple. For example, referring to FIG. 5, the information processing device (P) can detect two imaging areas (AR), a first imaging area (AR1) and a second imaging area (AR2), from elements (E1, E2) included in the acquired image frame (IM).

[0070] However, it is not limited to this, and the information processing device (P) may set the number of detected shooting areas (AR) differently depending on variables such as the curvature of one surface of a cylindrical pattern marker (M) included in an acquired image frame (IM), the number of elements included in the captured image, and the intensity of light that the elements can reflect.

[0071] The information processing device (P) extracts a first element (E1) and a second element (E2) from each of the multiple shooting areas (AR) detected from the image frame (IM), and maps the first elements (E1) included in the shooting area (AR) to a matching coordinate system based on an arrangement of the second elements (E2) included in the shooting area (AR), specifically a standard arrangement included in the shooting area (AR) (hereinafter, 'standard arrangement' can be interpreted as an arrangement of the second elements (E2) included in the image frame (IM) or the shooting area (AR) set in the image frame (IM)).

[0072] Referring to FIG. 5, when the first reference sequence section (150a) is an array in which the array is repeated at intervals of two first elements (E1) and the second reference sequence section (150b) is an array of first-second elements (E1-2), the information processing device (P) can detect three first elements (E1) arranged consecutively along the second direction from an image frame (IM) including a pattern section (100).

[0073] In this case, the shooting device (C) or the information processing device (P) can detect a "┗┛" shaped pattern from the first-second element (E1-2) of the reference sequence part (150) included in the image frame (IM), and can calculate the first shooting area (AR1) by detecting the arrangement of the second element (E2) of the unique sequence part (110) corresponding to the "┗┛" shape in the first direction.

[0074] As an optional embodiment, the imaging device (C) or information processing device (P) from the first-second element (E1-2) of the reference sequence section (150) included in the image frame (IM) " ㅗ It can detect a shape pattern, and the above " ㅗ A second shooting area (AR2) can be calculated by detecting the arrangement of the second element (E2) of the unique sequence part (110) corresponding to the shape and the first direction.

[0075] However, it is not limited to this, and depending on the number of reference sequence sections (150), the shape of the array in which the first element (E1) is repeated, etc., the shooting device (C) or the information processing device (P) can detect light of various shapes from the first element (E1) of the reference sequence section (150) on the shooting image, and can calculate the shooting area (AR) by detecting the array of the second element (E2) of the unique sequence section (110) corresponding to the detected light and the first direction.

[0076] For example, if the reference sequence section (150) is composed of four reference sequence sections (150) arranged sequentially along a first direction, and the arrangement of the four reference sequence sections (150) is sequentially composed of an arrangement consisting only of first-2 elements (E1-2), an arrangement in which one first-1 element (E1-1) and two first-2 elements (E1-2) are repeated, an arrangement in which one first-1 element (E1-1) and two first-2 elements (E1-2) are repeated, and an arrangement consisting only of first-2 elements (E1-2), then the imaging device (C) or the information processing device (P) [retrieves] from the first-2 element (E1-2) of the reference sequence section (150) on the image frame " grave It can detect shape patterns and calculate the augmented reality (AR) area based on this.

[0077] The information processing device (P) can detect various shapes from first elements (E1) included in the acquired image frame (IM) to calculate a shooting area (AR), and can map the first elements (E1) included in the calculated shooting area (AR) to a matching coordinate system.

[0078] Specifically, the information processing device (P) can determine an index value (idx) on a matching coordinate system to which the first elements (E1) included in the acquired image frame (IM) are mapped, based on an array of second elements (E2) included in a shooting area (AR) calculated from the acquired image frame (IM), and map the first elements (E1) to the determined index value (idx) on the matching coordinate system.

[0079] In the present specification, 'index value (idx, idx(1), id(2), idx(m))' can be interpreted as a coordinate value or position value on a matching coordinate system that corresponds one-to-one with the position information of the first element (E1) detected from the image of the first element (E1) included in the image frame.

[0080] In one embodiment, an information processing device (P) may be provided with a first image frame (IM1) in which an area of ​​a pattern marker (M) is captured at a first time point. In this case, the information processing device (P) may detect an arrangement of a first element (E1) included in the first image frame (IM1) and a second element (E2) corresponding in a second direction to the first element (E1) included in the first image frame (IM1).

[0081] Through this, the information processing device (P) can detect a first index value (idx 1) in which a first element (E1) included in the first image frame (IM1) can be mapped onto a matching coordinate system by using an array of second elements (E2) on the first image frame (IM1) detected above.

[0082] Furthermore, the information processing device (P) may receive a second image frame (IM2) in which the area of ​​the pattern marker is captured at a second time point different from the first time point. In this case, the information processing device (P) may detect an arrangement of a first element (E1) included in the second image frame (IM2) and a second element (E2) corresponding in a second direction to the first element (E1) included in the second image frame (IM2).

[0083] Through this, the information processing device (P) can detect a first index value (idx1) on a matching coordinate system corresponding to the position information of a first element (E1) included in the second image frame (IM2) by using the array of second elements (E2) on the second image frame (IM2) detected above.

[0084] Theoretically, when the time interval between the first time point and the second time point is set to be very small (e.g., 1 / 24 (sec)), the position information of the first element (E1) included in the first image frame (IM1) and the second image frame (IM2), respectively, in which the same area of ​​the pattern marker is captured, and the corresponding index value on the matching coordinate system will be the same.

[0085] However, due to variables in the shooting environment such as changes in illumination or unintentional vibrations, the position information of the first element (E1) detected by the information processing device (P) from image frames (IM) acquired at different times may have an error compared to the position information of the first element (E1) in the real coordinate system.

[0086] Accordingly, an information processing device (P) according to one embodiment of the present invention detects a first element (E1) included in one area of ​​a pattern marker multiple times at different times, and sequentially detects index values ​​(idx1, idx2) corresponding to the first elements (E1).

[0087] Through this, the information processing device (P) can calculate the mapping index value (idx(M)), which is an index value with a reduced error range, by using a preset algorithm on the index values ​​(idx1, idx2) detected sequentially.

[0088] A detailed description of the method for calculating a mapping index value (idx(M)) in which the error range of the first element (E1) is minimized by an information processing device (P) according to one embodiment of the present invention will be provided later.

[0089] Referring to FIG. 3, a unique sequence unit (110) according to one embodiment of the present invention may be composed of a plurality of second elements (E2) representing second information different from first information, and the plurality of second elements (E2) may be arranged along a second direction.

[0090] The second information may include ID information of the main body or the pattern marker (M). Specifically, when the main body is formed in a cylindrical shape with a length extended in the first direction, the second information may include ID information including attitude information of the pattern marker (M). The description regarding the ID information will be described in detail in the description regarding the main body in a cylindrical shape including the second element (E2) according to an embodiment of the present invention.

[0091] A second element (E2) according to one embodiment of the present invention may include a second-1 element (E2-1) and a second-2 element (E2-2) made of materials having different characteristics.

[0092] In one embodiment, when the pattern marker (M) is composed of a passive marker, the second-1 element (E2-1) and the second-2 element (E2-2) may be made of materials with different reflectances, and in this case, the second-2 element (E2-2) may be made of a material with a relatively higher light reflectance than the second-1 element (E2-1).

[0093] In an optional embodiment, if the pattern marker (M) is composed of an active marker, the 2-1 element (E2-1) may be made of a material that does not emit light, and the 2-2 element (E2-2) may be made of a light source capable of emitting light.

[0094] As a result, the external imaging device (C) can detect light from the second-2 element (E2-2), but cannot detect light from the second-1 element (E2-1), so it can detect the arrangement of the unique sequence unit (110) and the second information corresponding thereto according to the light pattern detected from the second-1 element (E2-1).

[0095] Referring to FIG. 4, when an external imaging device (C) detects light from a second element (E2), the information processing device (P) recognizes the second element (E2) in which light is detected as "1", and if no light is detected, it recognizes it as "0". In one embodiment, the information processing device (P) recognizes the second-2 element (E2-2) as "1" and the second-1 element (E2-1) as "0".

[0096] Referring to FIG. 3, a unique sequence unit (110) according to one embodiment of the present invention may include at least one encoding sequence unit (120), and the encoding sequence unit (120) may consist of a second-1 element (E2-1) and a second-2 element (E2-2) arranged according to second information.

[0097] In one embodiment, at least one encoding sequence unit (120) may be provided with a first encoding sequence unit (120a) and a second encoding sequence unit (120b), and the first encoding sequence unit (120a) and the second encoding sequence unit (120b) may be arranged sequentially along a second direction.

[0098] In one embodiment, each of at least one encoding sequence unit (120) may include an equal number of second elements (E2). For example, if the second encoding sequence unit (120b) is composed of eight second elements (E2) that are recognized by the information processing device (P) as an array of "00011101", the first encoding sequence unit (120a) may be composed of eight second elements (E2) that are recognized by the information processing device (P) as an array of "00001111". As a result, when the main body is formed such that the area where the first second element (E2) in the second direction of the first encoding sequence part (120a) is located and the area where the last second element (E2) is located are adjacent, and the area where the first second element (E2) in the second direction of the second encoding sequence part (120b) is located and the area where the last second element (E2) is located are adjacent, the shape of the main body can be formed as a cylinder.

[0099] In one embodiment, the number of second elements (E2) provided by each of at least one encoding sequence unit (120) may be configured to be equal to the number of first elements (E1) provided by the unique sequence unit (110). As a result, when the main body is formed such that the area where the first second element (E2) in the second direction of the encoding sequence unit (120) is located and the area where the last second element (E2) is located are adjacent, and the area where the first first element (E1) in the second direction of the unique sequence unit (110) is located and the area where the last first element (E1) is located are adjacent, the shape of the main body may be formed in a cylindrical shape.

[0100] Hereinafter, a cylindrical shape of a main body forming the appearance of a pattern marker (M) or a pattern marker (M) according to one embodiment of the present invention will be described in detail.

[0101] In this specification, the term "cylindrical" is not limited to a tube shape with a circular cross-section, but can be interpreted as a tube shape having a cross-section of various shapes, such as elliptical or polygonal.

[0102] In one embodiment, the number of first elements (E1) forming the second direction arrangement of the reference sequence section (150) and the number of second elements (E2) forming the second direction arrangement of the unique sequence section (110) may be the same. For example, the number of elements (E1, E2) forming the reference sequence section (150) and the unique sequence section (110) along the second direction may each be provided as N.

[0103] Hereinafter, a cylindrical shape of a pattern marker (M) is described in which the number of elements (E1, E2) forming the second direction arrangement of the reference sequence part (150) and the unique sequence part (110) are each provided as N.

[0104] In one embodiment, N elements may be arranged along the circumferential direction of a cylindrical pattern marker (M). In this case, the unique sequence portion (110) included in different shooting areas (AR) may be arranged in an array of 2-1 elements (E2-1) and 2-2 elements (E2-2) such that they do not overlap within a 360° range with respect to the longitudinal central axis of the cylindrical pattern marker (M).

[0105] Specifically, any one standard arrangement formed by the second-1 element (E2-1) and the second-2 element (E2-2) that is pre-set on the outer surface of the cylindrical pattern marker (M) may be arranged in a second direction such that the second-1 element (E2-1) and the second-2 element (E2-2) do not overlap with all standard arrangements that can be set within a 360° range with respect to the longitudinal central axis of the cylindrical pattern marker (M).

[0106] For example, if the number of elements (E1, E2) forming the second direction array of the reference sequence section (150) and the unique sequence section (110) is N each, the number of standard arrays that can be set within a 360° range based on the longitudinal central axis of the cylindrical pattern marker (M), regardless of the size of the standard array, is N. In this case, the arrays of the 2-1 element (E2-1) and the 2-2 element (E2-2) corresponding to the N standard arrays do not overlap with each other.

[0107] Referring to FIGS. 5 and 6, when the size of the pattern portion (100) included in the image frame (IM) obtained from the imaging device (C) or the size of the imaging device is set to A × 3 (hereinafter, 'A' is defined as the number of elements arranged in the first direction of the pattern portion (100)), a standard arrangement that can be detected within a 360° range based on the longitudinal central axis of the cylindrical pattern marker (M) can be arranged as follows.

[0108]

[0109]

[0110] In an optional embodiment, when the size of the pattern portion (100) included in the image frame (IM) obtained from the imaging device (C) or the size of the imaging device is set to A x 4, a standard arrangement that can be detected within a 360° range based on the longitudinal central axis of the cylindrical pattern marker (M) may be listed as follows.

[0111] ,

[0112]

[0113] In this way, the second element (E1) can be arranged so that each standard array that can be set or extracted from a single pattern marker (M) does not overlap with each other, and furthermore, the first second element (E2) and the last second element (E2) in the second direction of the cylindrical pattern marker (M) can be set so that each standard array that can be extracted does not overlap with each other.

[0114] As a result, as the cylindrical pattern marker (M) rotates, the standard arrays included in the image frame (IM), each containing different regions of the cylindrical pattern marker (M), correspond one-to-one (1:1) with the rotation angle of the cylindrical pattern marker (M). Through this, the information processing device (P) can map the first element (E1) included in the image frame (IM) to an index value in a matching coordinate system that corresponds one-to-one (1:1) with the standard array included in the image frame (IM).

[0115] Below is a technical description regarding the arrangement of the second element (E2) of the unique sequence part (110).

[0116] The unique sequence section (110) may include a plurality of second elements (E2) arranged at predetermined intervals along a second direction. Specifically, the unique sequence section (110) may include at least one encoding sequence section (120) arranged in parallel along a first direction, and each of the at least one encoding sequence section (120) may include a plurality of second elements (E2) arranged at predetermined intervals along a second direction.

[0117] The second elements (E2) of the encoding sequence unit (120) according to one embodiment of the present invention may be composed of binary codes.

[0118] Hereinafter, a detailed description of the M-order binary code described in this specification is provided.

[0119] The 3rd binary code array is "00011101" which can sequentially recognize "000", "001", "011", "111", "110", "101", "010", and "100", corresponding to three-digit binary numbers.

[0120] The 4th binary code array is "0000111101100101" which can sequentially recognize "0000", "0001", "0011", "0111", "1111", "1110", "1101", "1011", "0110", "1100", "1001", "0010", "0101", "1010", "0100", "1000".

[0121] In the encoding sequence section (120), a plurality of second elements (E2) can be arranged in an M-order binary code array, and at least one encoding sequence section (120) can be arranged in binary code arrays of different orders.

[0122] In one embodiment, M may be a natural number greater than or equal to 3.

[0123] In one embodiment, the first encoding sequence part (120a) may follow a fourth binary code array, and the second encoding sequence part (120b) may follow a third binary code array.

[0124] In this case, as illustrated in FIG. 3, the first encoding sequence section (120a) is arranged such that the second elements (E2) are arranged in an array of “0000111101100101” along the second direction, and the second-1 element (E2-1) is located in the area corresponding to “0” and the second-2 element (E2-2) is located in the area corresponding to “1”.

[0125] Additionally, as illustrated in FIG. 3, the second encoding sequence section (120b) may have the second elements (E2) arranged in an array of “00011101” along the second direction, where the second-1 element (E2-1) is located in the area corresponding to “0” and the second-2 element (E2-2) is located in the area corresponding to “1”.

[0126] However, the number of second elements (E2) may be limited depending on the size of the pattern marker (M), etc. In one embodiment, when the first encoding sequence unit (120a) follows a 4th binary code array but is limited to 8 second elements (E2), the plurality of second elements (E2) may follow an array of “00001111”.

[0127] Referring to FIGS. 6 to 8, a main body according to one embodiment of the present invention may be formed in a cylindrical shape with a length extended in a first direction. Furthermore, the second direction may be the circumferential direction of the main body.

[0128] The pattern portion (100) may be formed in a cylindrical shape so that the first element of the second direction (the term collectively referred to as the first to third elements (E1, E2, E3) is defined as the element) and the last elements are arranged adjacently.

[0129] Specifically, the reference sequence section (150) may have a circular or cylindrical shape such that the area where the first first element (E1) of the second direction is located and the area where the last first element (E1) is located are adjacent to each other.

[0130] As a result, there is a unique effect in that the imaging device (C) and the information processing device (P) can detect the first element (E1) of the reference sequence part (150) regardless of the angle formed by the longitudinal center axis of the cylindrical pattern marker (M) where the pattern part (100) is formed and the X-axis, Y-axis, and Z-axis (see FIG. 8) respectively, and the rotation angle based on the longitudinal center axis of the cylindrical pattern marker (M).

[0131] In addition, when the second reference sequence section (150b) is composed of an array of first-second elements (E1-2) that are continuous in the second direction, the imaging device (C) and the information processing device (P) can detect light from the continuous first-second elements (E1-2) regardless of the angle formed by the longitudinal center axis of the cylindrical pattern marker (M) and the X-axis, Y-axis, and Z-axis (see FIG. 8) respectively, and the rotation angle based on the longitudinal center axis of the cylindrical pattern marker (M). Therefore, there is a unique effect of being able to detect second information regarding the orientation of the pattern marker (M), such as the second direction or the circumferential direction of the cylindrical pattern marker (M), from the detected light.

[0132] Through this, the imaging device (C) can detect the first elements (E1) regardless of the rotation angle based on the longitudinal central axis of the cylindrical pattern marker (M), so the detected first elements (E1) can be mapped to the index value of a matching coordinate system corresponding to a standard array that corresponds one-to-one with the rotation angle based on the longitudinal central axis of the cylindrical pattern marker (M).

[0133] In addition, when the first elements (E1) of the first reference sequence section (150a) are formed as an array of secondary binary codes and the second reference sequence section (150b) is formed as an array of first-second elements (E1-2) that are continuous in the second direction, regardless of the angles formed by the longitudinal center axis of the cylindrical pattern marker (M) and the X-axis, Y-axis, and Z-axis (see FIG. 8), respectively, and the rotation angle based on the longitudinal center axis of the cylindrical pattern marker (M), a "┗┛" shaped pattern and " ㅗ Since the pattern of the shape can be detected, the shooting device (C) and the information processing device (P) have the effect of being able to calculate the shooting area (AR) from the light of a specific shape detected from the reference sequence unit (150).

[0134] The unique sequence section (110) may have a circular or cylindrical shape such that the area where the first second element (E2) of the second direction is located and the area where the last second element (E2) is located are adjacent to each other.

[0135] As a result, there is a unique effect in that the imaging device (C) and the information processing device (P) can detect the second element (E2) of the unique sequence part (110) regardless of the angle formed by the longitudinal center axis of the cylindrical pattern marker (M) where the pattern part (100) is formed and the X-axis, Y-axis, and Z-axis (see FIG. 8) respectively, and the rotation angle based on the longitudinal center axis of the cylindrical pattern marker (M).

[0136] When the unique sequence section (110) follows a binary code array of different orders, the array of each second element (E2) included in the image frames (IM) in which different areas of the pattern marker (M) are captured within a 360° range based on the longitudinal central axis of the cylindrical pattern marker (M) cannot be duplicated.

[0137] Accordingly, the image frame (IM) acquired by the imaging device (C) according to the rotation angle based on the longitudinal central axis of the cylindrical pattern marker (M) of the pattern marker (M) includes an array of second elements (E2) that correspond one-to-one with said rotation angle, and the information processing device (P) can recognize the array of second elements (E2) that correspond one-to-one as pattern marker (M) ID information.

[0138] In conclusion, since the information processing device (P) can determine the area of ​​the captured pattern marker (M) from the array of second elements included in the image frame (IM), there is a unique effect of being able to map the first element (E1) included in the image frame (IM) to an index value on a matching coordinate system corresponding to the area of ​​the captured pattern marker (M).

[0139] The section of the separation sequence (170) may have a circular or cylindrical shape such that the area where the first third element (E3) of the second direction is located and the area where the last third element (E3) is located are adjacent to each other.

[0140] Hereinafter, a method for an information processing device to receive multiple image frames captured at different points in time and map detected pattern parts to a matching coordinate system is described in detail.

[0141] The chronological description of the method in this specification is for convenience of explanation, and the chronological order of the method and steps described below is not limited to the order described in this specification and may be performed simultaneously.

[0142] The information processing device (P) may include one or more of a main processor, an auxiliary processor, or memory. It may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if an electronic device includes a main processor and an auxiliary processor, the auxiliary processor may be configured to use lower power than the main processor or to be specialized for a specified function. The auxiliary processor may be implemented separately from the main processor or as part thereof.

[0143] An auxiliary processor can control at least some of the functions or states associated with at least one component of an electronic device (e.g., a display module, a sensor module, or a communication module), for example, on behalf of the main processor while the main processor is in an inactive (e.g., sleep) state, or together with the main processor while the main processor is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (e.g., an image signal processor or a communication processor) may be implemented as part of another functionally related component (e.g., a camera module or a communication module). According to one embodiment, the auxiliary processor (e.g., a neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device itself where the artificial intelligence is performed, or through a separate server (e.g., a server). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above.Artificial intelligence models may include software structures, either additionally or as a substitute, in addition to hardware structures.

[0144] According to various embodiments, the imaging device (C) may include an imaging camera (e.g., RGBD camera) for capturing an image corresponding to the user's field of view (FoV) and / or measuring the distance to an object, an eye tracking camera module for verifying the direction of the user's gaze, and / or a gesture camera module for recognizing a certain space. According to one embodiment, the imaging device (C) may include a high-resolution camera such as an HR (high resolution) camera and / or a PV (photo video) camera.

[0145] A shooting device (C) or an information processing device (P) can generate an image frame (IM) by shooting a preset area of ​​a pattern section (100) that includes a plurality of elements (E1, E2).

[0146] The shooting device (C) can capture the pattern marker (M) at a preset frame rate, and an image frame (IM) formed on the lens per second can be generated to correspond to the frame rate.

[0147] When the preset frame rate of the shooting device (C) is N fps, an image frame (IM) of a pattern marker (M) can be formed N times per second on the lens equipped with the shooting device (C), and N image frames (IM) are generated per second. As a result, the information processing device (P) generates N image frames (IM) per second that include a region of the unique sequence section (110) and the reference sequence section (150).

[0148] Referring to FIG. 7, the information processing device (P) may receive multiple image frames (IM) captured at different times from a shooting device (C) that captures a pattern marker (M).

[0149] Specifically, the information processing device (P) may receive a first image frame (IM1) in which an area of ​​the pattern marker (M) is captured at a first time point from the shooting device (C), and furthermore, the information processing device (P) may receive a second image frame (IM2) in which an area of ​​the pattern marker (M) is captured at a second time point different from the first time point from the shooting device (C).

[0150] In this specification, only the first image frame (IM1) generated by being captured at a first time point and the second image frame (IM2) generated by being captured at a second time point are described, but this is merely to facilitate the explanation of the process of fusing information obtained from image frames (IM) generated by being captured at different times point, and the capturing device (C) can continuously capture an area of ​​the pattern marker (M) at a preset time interval, and the information processing device (P) can acquire two or more image frames (IM) in which an area of ​​the pattern marker (M) is captured.

[0151] Furthermore, as the information processing device (P) acquires a large number of image frames (IM) in which a region of the pattern marker (M) is captured, the reliability of the mapping index (idx(M)) value on the matching coordinate system in which the first element included in the region of the pattern marker (M) is mapped can be improved.

[0152] When an information processing device (P) maps a pattern part (100) included in an image frame (IM) captured at a specific point in time to a matching coordinate system, there was a problem in that the position information of the pattern part (100) or elements (E1, E2) could not be accurately detected due to shooting errors caused by reasons such as vibration in the shooting environment or changes in illumination.

[0153] As a result, there was a problem in that, in the process where the information processing device (P) maps the pattern part (100) or elements detected at the specific time point to a virtual matching coordinate system, the accuracy of the index value on the matching coordinate system to which the pattern part (100) or elements are mapped cannot be determined, and thus it is not easy to predict the accuracy of the position and orientation of the pattern marker (M) calculated by the information processing device (P).

[0154] Specifically, the center point of the pattern marker (M) must be determined by observing the first elements (E1) arranged along the circumferential direction of the cylindrical pattern marker (M) and mapping the observed values ​​of each first element (E1) to appropriate index values ​​of the matching coordinate system. However, due to shooting errors of the shooting device (C), it was not possible to verify whether an error occurred in the index values ​​of the matching coordinate system to which the first elements (E1) are mapped. Consequently, there was a problem in that the reliability of the position and orientation of the pattern marker (M) could not be secured using the inaccurate center point of the pattern marker (M) calculated by the information processing device (P).

[0155] In one embodiment, the time difference between the first time point and the second time point may be 1 / N (sec) ('N' is the frame rate per second (fps) of the imaging device (C), for example, N=24), and in this case, the second image frame (IM2) may be an image frame (IM) generated immediately after the first image frame (IM1).

[0156] Accordingly, the position value of the first element (E1(IM1)) in the world coordinate system at the first time point and the position value of the first element (E1(IM2)) in the world coordinate system at the second time point can be very close, and an information processing device according to an embodiment of the present invention will explain in detail how to solve the above problem by using this.

[0157] A shooting device (C) according to one embodiment of the present invention can continuously shoot an area of ​​the same pattern portion (100) of a pattern marker (M) at a preset time interval, and an information processing device (P) can acquire a plurality of image frames (IM) including an area of ​​the same pattern portion (100).

[0158] Referring to FIGS. 7 and FIGS. 9, an information processing device (P) according to one embodiment of the present invention can detect a pattern portion (100) of a pattern marker (M) from each of a plurality of image frames (IM).

[0159] In one embodiment, the information processing device (P) can detect a first element (E1(IM1)) included in a first image frame (IM1) and a first element (E1(IM2)) included in a second image frame (IM2).

[0160] In one embodiment, the information processing device (P) can detect an array of second elements (E2(IM1)) corresponding to a first element (E1(IM1)) included in a first image frame (IM1), and can detect an array of second elements (E2(IM2)) corresponding to a first element (E1(IM2)) included in a second image frame (IM2).

[0161] Specifically, the information processing device (P) can detect the arrangement of a first element (E1(IM1)) included in a first image frame and a second element (E2(IM1)) adjacent in a first direction to calculate position information of the first element (E1(IM1)) included in the first image frame (IM1) on the outer surface of a cylindrical pattern marker (M), and can detect the arrangement of a first element (E1(IM2)) included in a second image frame (IM2) and a second element (E2(IM2)) adjacent in a first direction to calculate position information of the first element (E1(IM2)) included in the second image frame (IM2) on the outer surface of a cylindrical pattern marker (M).

[0162] In one embodiment, the information processing device (P) can detect a first index value (idx1) on a matching coordinate system by using an array of second elements (E2(IM1)) corresponding to a first element (E1(IM1)) included in a first image frame (IM1).

[0163] In this specification, 'first index value (idx1)' can be interpreted as the index value (idx) on the alignment coordinate system of the first element (E1) calculated by the information processing device (P), assuming that the position of the first element (E1(IM1)) detected from the first image frame (IM1) or the position of light detected from the first image frame (IM1) is identical to the actual position of the first element (E1).

[0164] The information processing device (P) can detect a second index value (idx2) on a matching coordinate system by using an array of second elements (E2(IM2)) corresponding to a first element (E1(IM2)) included in the second image frame (IM2).

[0165] In the present specification, the second index value (idx2) can be interpreted as the index value (idx) on the alignment coordinate system of the first element (E1) calculated by the information processing device (P), assuming that the position of the first element (E1(IM2)) detected from the second image frame (IM2) or the position of light detected from the second image frame (IM2) is identical to the actual position of the first element (E1).

[0166] In one embodiment, the first index value (idx1) and the second index value (idx2) may follow a preset probability distribution. For example, the first index value (idx1) and the second index value (idx2) may follow a Gaussian probability distribution.

[0167] The information processing device (P) can calculate the weight of the first index value (idx1) and the weight of the second index value (idx2) using a preset probability distribution.

[0168] Specifically, the information processing device (P) calculates the difference between the expected value of a preset probability distribution and the first index value (idx1), and if the difference is large, it can assign a small weight to the first index value (idx1), and if the difference is small, it can assign a large weight to the first index value (idx1).

[0169] Additionally, the information processing device (P) calculates the difference between the expected value of the preset probability distribution and the second index value (idx2), and if the difference is large, it may assign a small weight to the second index value (idx2), and if the difference is small, it may assign a large weight to the second index value (idx2).

[0170] An information processing device (P) according to one embodiment of the present invention can calculate a mapping index value (idx(M)) in which a first element (E1) included in an area where a pattern marker (M) is captured is mapped on a matching coordinate system using a first index value (idx1), a weight of the first index value (idx1), a second index value (idx2), a weight of the second index value (idx2), and a preset algorithm.

[0171] Specifically, the information processing device (P) can calculate the probability distribution of the index value (idx) detected by photographing a region of the pattern marker (M) at different times using a first index value (idx1), a weight of the first index value (idx1), a second index value (idx2), a weight of the second index value (idx2), and a preset algorithm, and can derive the expected value of the calculated probability distribution as a mapping index value (idx(M)).

[0172] In one embodiment, the preset algorithm may be a linear Kalman filter. Specifically, based on the linear Kalman filter, the expected or estimated value of the mapping index value (idx(M)) of the first element (E1) can be calculated using observations of multiple first elements (E1) obtained in the past.

[0173] In the present invention, a mapping index value (idx(M)) of the first element (E1) can be calculated using a linear Kalman filter and the first index value (idx1), the second index value (idx2), and the nth index value obtained in the past.

[0174] In this specification, the 'n-th index value' can be interpreted as an index value on the alignment coordinate system of the first element (E1) calculated by the information processing device (P), provided that the position of the first element detected from the image frame (IM(n)) acquired at the n-th time point or the position of light detected from the n-th image frame (IM(n)) is assumed to be identical to the actual position of the first element (E1).

[0175] Additionally, in this specification, the 'mapping index value (idx(M))' can be interpreted as the index value of a first element (E1) that the information processing device (P) maps to a matching coordinate system in order to calculate the coordinate system of the pattern marker (M) and the position and orientation of the pattern marker (M).

[0176] Hereinafter, a method for an information processing device (P) to calculate a first index value (idx1) to an nth index value, a method for obtaining a probability distribution that follows a plurality of index values ​​(idx) using the index values ​​(idx), and a method for obtaining a mapping index value (idx(M)) will be described in detail.

[0177] Referring to FIGS. 7 and 9, an information processing device (P) according to one embodiment of the present invention receives a first image frame (IM1) in which a region of a pattern marker (M) is captured at a first time point and can detect a first element (E1(IM1)) included in the region of the pattern marker (M) and a second element (E2(IM1)) corresponding to the first element (E1(IM1)).

[0178] Specifically, the information processing device (P) can detect an array of second elements (E2(IM1)) corresponding to the first element (E1(IM1)), and since the array of second elements (E2(IM1)) includes second information, the information processing device (P) can calculate position information on the outer surface of the cylindrical pattern marker (M) of the first element (E1(IM1)) and second element (E2(IM1)) included in the first image frame (IM1).

[0179] In addition, the information processing device (P) can calculate information regarding the positional relationship between a plurality of first elements (E1(IM1)) in a matching coordinate system by using the transformation relationship between the world coordinate system and the coordinate system of the shooting device (C) and the image information of a plurality of first elements (E1(IM1)) included in the first image frame (IM1).

[0180] Through this, the information processing device (P) can detect a first index value (idx1) in which the first elements (E1(IM1)) included in the first image frame (IM1) are mapped to a matching coordinate system by utilizing the arrangement of second elements (E2(IM1)) included in the first image frame (IM1) and the positional relationship between the first elements (E1(IM1)).

[0181] The method by which the information processing device (P) detects the second index value (idx2) and the nth index value from the second image frame (IM2) or the nth image frame (IM) is the same as the method of detecting the first index value (idx1) from the first image frame (IM1), so a description related thereto is omitted.

[0182] In one embodiment, the index value of a first element (E1) detected from an image frame (IM) in which a region of a pattern marker (M) is captured follows a preset probability distribution, and subsequently, an information processing device (P) can calculate a newly adjusted probability distribution by fusing a newly observed and detected first index value (idx1) and a weight of the first index value (idx1) with the preset probability distribution using a preset algorithm.

[0183] As described above, the information processing device (P) can calculate a readjusted probability distribution by fusing the newly observed and detected second index value (idx2) and the weight of the second index value (idx2) with the newly adjusted probability distribution using a preset algorithm.

[0184] Additionally, the information processing device (P) can readjust the probability distribution by fusing the n-th index value and the weight of the n-th index value to the readjusted probability distribution using a preset algorithm, and as n accumulates, the error covariance of the adjusted probability distribution decreases.

[0185] In conclusion, as the observed value of the first element (E1) that is pre-set accumulates, the error covariance of the probability distribution following the index values ​​on the matching coordinate system of the first element (E1) captured by the shooting device (C) decreases, and as a result, as the observed value of the first element (E1) accumulates, the reliability of the mapping index value (idx(M)) of the captured first element (E1) detected by the information processing device (P) increases.

[0186] Accordingly, when the information processing device (P) receives multiple image frames (IM) in which the shooting device (C) has captured the area of ​​the preset pattern portion (100) of the pattern marker (M) multiple times, the reliability of the mapping index value (idx(M)) on the alignment coordinate system of the first element (E1) included in the preset area is increased, and accordingly, the reliability of the center point of the pattern marker (M) calculated from the mapping index values ​​(idx(M)) of the first elements on the alignment coordinate system is also increased.

[0187] This has the effect of increasing the reliability of the coordinate system of the pattern marker (M) calculated as the center point of the pattern marker (M), and the position and orientation of the pattern marker (M) derived from the coordinate system of the pattern marker (M).

[0188] The scope of the present invention is not limited to the embodiments described above, and all scopes equivalent to or equivalently modified from the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the concept of the present invention. Explanation of the symbols

[0189] M: Pattern Marker 100: Pattern Section 110: Unique sequence section 150: Reference sequence section IM: Image frame IDX: Index value

Claims

Claim 1 A step of receiving a plurality of image frames captured at different times from a capturing device for capturing a pattern marker; a step of detecting a pattern portion of the pattern marker from each of the plurality of image frames; and a step of mapping the detected pattern parts to a matching coordinate system; wherein the step of detecting the pattern parts includes detecting a reference sequence part representing first information and a unique sequence part representing second information different from the first information from the pattern parts, and the step of mapping to the matching coordinate system includes mapping the detected reference sequence part to an index value on the matching coordinate system corresponding to the second information of the detected unique sequence part, wherein the reference sequence part has a plurality of first elements and the unique sequence part has a plurality of second elements, wherein the reference sequence part and the unique sequence part are sequentially arranged along a first direction of the pattern marker, wherein each of the plurality of first elements and the plurality of second elements is arranged along a second direction different from the first direction, and the arrangement of the plurality of second elements follows a de Bruijn sequence, and the step of receiving the plurality of image frames includes a step of receiving a first image frame in which a region of the pattern marker is captured at a first time point; The method includes the step of receiving a second image frame in which a portion of the pattern marker is captured at a second time point different from the first time point; wherein the step of detecting the pattern portion comprises: a step of detecting the first element included in the first image frame; a step of detecting an array of the second element corresponding to the first element included in the first image frame; a step of detecting the first element included in the second image frame; and a step of detecting the array of the second element corresponding to the step of detecting the first element included in the second image frame.A pattern marker mapping method comprising: a step of detecting a first element included in a first image frame, wherein the step of detecting a first element included in the first image frame includes a step of detecting a first index value on the matching coordinate system using an array of second elements corresponding to the first element included in the first image frame; and a step of detecting a first element included in the second image frame includes a step of detecting a second index value on the matching coordinate system using an array of second elements corresponding to the first element included in the second image frame; wherein the first index value and the second index value follow a preset probability distribution. Claim 2 delete Claim 3 A pattern marker mapping method according to claim 1, wherein the pattern marker is a three-dimensional shape extending in the first direction, the first direction is parallel to the longitudinal central axis of the pattern marker, and the second direction is the circumferential direction of the pattern marker. Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 A pattern marker mapping method according to claim 1, characterized in that the preset probability distribution is a Gaussian distribution. Claim 8 A pattern marker mapping method according to claim 1, wherein the step of mapping to the matching coordinate system comprises: a step of calculating the weight of the first index value and the weight of the second index value using the preset probability distribution; and a step of calculating the index value to which the first element included in the one region of the pattern marker is mapped onto the matching coordinate system using the first index value, the weight of the first index value, the second index value, the weight of the second index value, and a preset algorithm. Claim 9 A pattern marker mapping method according to claim 8, characterized in that the preset algorithm is a linear Kalman filter. Claim 10 A three-dimensional pattern marker having a pre-set pattern formed around its perimeter; a shooting device for capturing the pattern marker; and an information processing device, wherein the information processing device comprises the steps of: receiving a plurality of image frames captured at different times from the shooting device for capturing the pattern marker; and detecting a pattern portion of the pattern marker from each of the plurality of image frames. and performing the step of mapping the detected pattern parts to a matching coordinate system; wherein the step of detecting the pattern parts involves detecting a reference sequence part representing first information and a unique sequence part representing second information different from the first information from the pattern parts; the step of mapping to the matching coordinate system involves mapping the detected reference sequence part to an index value on the matching coordinate system corresponding to the second information of the detected unique sequence part; wherein the reference sequence part has a plurality of first elements and the unique sequence part has a plurality of second elements; wherein the reference sequence part and the unique sequence part are sequentially arranged along a first direction of the pattern marker, wherein each of the plurality of first elements and the plurality of second elements is arranged along a second direction different from the first direction, and the arrangement of the plurality of second elements follows a de Bruijn sequence; and the step of receiving the plurality of image frames includes the step of receiving a first image frame in which an area of ​​the pattern marker is captured at a first time point; The method includes the step of receiving a second image frame in which a portion of the pattern marker is captured at a second time point different from the first time point; wherein the step of detecting the pattern portion comprises: a step of detecting the first element included in the first image frame; a step of detecting an array of the second element corresponding to the first element included in the first image frame; and a step of detecting the first element included in the second image frame.A pattern marker mapping system comprising: a step of detecting an array of the second elements corresponding to a step of detecting the first element included in the second image frame; wherein the step of detecting the first element included in the first image frame includes a step of detecting a first index value on the matching coordinate system using an array of the second elements corresponding to the first element included in the first image frame; and wherein the step of detecting the first element included in the second image frame includes a step of detecting a second index value on the matching coordinate system using an array of the second elements corresponding to the first element included in the second image frame; and wherein the first index value and the second index value follow a preset probability distribution.

Images