Augmented reality device for acquiring three-dimensional position

Abstract

An augmented reality (AR) apparatus for obtaining three-dimensional (3D) position information of a hand joint from a plurality of images obtained by a plurality of cameras and a method of operating the same are disclosed. The AR device may obtain 2D joint coordinate values for feature points of a hand joint from a plurality of images obtained by photographing a user's hand by a plurality of cameras, estimate 3D joint coordinate values of the hand joint based on 2D joint coordinate values obtained from image combinations each composed of at least two of the plurality of images, and determine the 3D joint coordinate values of the hand joint based on the 3D joint coordinate values. An image combination with the minimum calculated error distance is selected from among the image combinations, and 3D position information of the hand joint is obtained based on a combination of 2D joint coordinate values from at least two images constituting the selected image combination.

Description

Technical Field

[0001] This disclosure relates to an augmented reality (AR) device and its operating method for obtaining three-dimensional (3D) position information of a user's hand joints. Specifically, this disclosure provides an AR device and its operating method for obtaining 3D position coordinate information of a user's hand joints from two-dimensional (2D) images obtained by using multiple cameras to capture images of the user's hand. Background Technology

[0002] Augmented Reality (AR) is a technology used to display virtual images by overlaying them onto the physical environment or objects of the real world. AR devices (such as smart glasses) are being used effectively in daily life for purposes such as information retrieval, navigation, camera shooting, and gaming. Specifically, smart glasses can also be worn as fashion items and are primarily used for outdoor activities.

[0003] For AR devices to provide AR services, hand interaction using the user's three-dimensional (3D) pose and gestures as input is crucial for the input interface. For example, AR services can provide a user interface that utilizes interactions made through the user's hands (e.g., selecting menu elements, interacting with virtual objects, selecting items, or placing objects on a virtual hand). Therefore, a technology is needed to obtain 3D positional information of the hand joints, accurately track the hand's pose (shape) using this 3D positional information, and recognize gestures to achieve more realistic AR technology.

[0004] AR devices typically use vision-based hand tracking technology to recognize the user's hand from images captured by a camera mounted on the AR device, freeing up the user's hands without the need for a separate external input device. AR devices obtain 3D positional information of the hand joints using triangulation based on multiple 2D images obtained by using a stereo camera (comprising two or more cameras) in an overlapping area between fields of view, and the positional relationship between the cameras; or they obtain 3D positional information of the hand joints using a "ray intersection" method, which estimates 3D positional coordinates based on the intersection point of virtual rays extending from the center positions of multiple cameras to the 2D positional coordinates of the joints in the multiple 2D images.

[0005] To obtain 3D positional information of hand joints from multiple 2D images, the accuracy of the 2D positional coordinates of feature points of the hand joints identified from these images is crucial. Inaccurate 2D positional coordinates obtained from some of the multiple 2D images can increase the error in the 3D positional information and reduce its accuracy. Furthermore, for typical RGB cameras, due to lens characteristics, 2D images may be distorted at the edges of the entire image area, and distortion errors may occur during image correction. Distortion errors caused by 2D image distortion can lead to errors in the obtained 3D positional information of the hand joints and reduce its accuracy.

[0006] When the accuracy of the 3D position information of the hand joints is low, AR devices may fail to recognize or misrecognize hand postures or gestures. Summary of the Invention

[0007] Solution to the problem

[0008] One aspect of this disclosure provides a method for an augmented reality (AR) device to obtain three-dimensional (3D) position information of a hand joint. According to embodiments of this disclosure, an operating method of an AR device may include: obtaining two-dimensional (2D) joint coordinate values ​​for feature points of the hand joint from multiple images obtained by capturing a user's hand through multiple cameras. The operating method of the AR device may include: estimating 3D joint coordinate values ​​of the hand joint based on the 2D joint coordinate values ​​obtained from at least two images constituting a plurality of images. The operating method of the AR device may include: selecting an image combination with the smallest error distance from the image combination, the error distance being calculated based on the estimated 3D joint coordinate values. The operating method of the AR device may include: obtaining 3D position information of the hand joint based on a combination of 2D joint coordinate values ​​from at least two images constituting the selected image combination.

[0009] Another aspect of this disclosure provides an AR device for obtaining 3D positional information of a hand joint. According to embodiments of this disclosure, the AR device may include a plurality of cameras for acquiring multiple images by photographing a user's hand, a memory storing at least one instruction, and at least one processor configured to execute the at least one instruction. The at least one processor may execute at least one instruction to obtain 2D joint coordinate values ​​for feature points of the hand joint from the multiple images acquired by the plurality of cameras. The at least one processor may execute at least one instruction to estimate 3D joint coordinate values ​​of the hand joint based on a combination of 2D joint coordinate values ​​obtained from at least two images constituting the plurality of images. The at least one processor may execute at least one instruction to select the image combination with the smallest error distance from the image combination, the error distance being calculated based on the estimated 3D joint coordinate values. The at least one processor may execute at least one instruction to obtain 3D positional information of the hand joint based on a combination of 2D joint coordinate values ​​from at least two images constituting the selected image combination.

[0010] Another aspect of this disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions readable by an AR device to cause the AR device to perform the following operations: obtaining 2D joint coordinate values ​​for feature points of a hand joint from multiple images obtained by capturing a user's hand through multiple cameras; estimating 3D joint coordinate values ​​of the hand joint based on a combination of 2D joint coordinate values ​​obtained from at least two of the multiple images; selecting the image combination with the smallest error distance from the image combination, the error distance being calculated based on the estimated 3D joint coordinate values; and obtaining 3D position information of the hand joint based on a combination of 2D joint coordinate values ​​from at least two images constituting the selected image combination. Attached Figure Description

[0011] This disclosure can be readily understood by taking into account the following detailed description and accompanying drawings, where reference numerals denote structural elements.

[0012] Figure 1 This is a conceptual diagram illustrating the operation of an augmented reality (AR) device for obtaining three-dimensional (3D) position information of a hand joint, according to embodiments of the present disclosure.

[0013] Figure 2 This is a flowchart illustrating a method for an AR device to obtain 3D position information of a hand joint according to an embodiment of the present disclosure.

[0014] Figure 3 This is a block diagram illustrating the components of an AR device according to an embodiment of the present disclosure.

[0015] Figure 4This is a flowchart illustrating a method by which an AR device according to an embodiment of the present disclosure obtains two-dimensional (2D) joint coordinate values ​​for feature points of a hand joint from multiple images and estimates 3D joint coordinate values ​​of the hand joint based on the obtained 2D joint coordinate values.

[0016] Figure 5 This is a flowchart illustrating an AR device according to an embodiment of the present disclosure selecting a combination of 2D joint coordinate values ​​based on an error distance calculated based on the estimated 3D joint coordinate values.

[0017] Figure 6 This is a flowchart illustrating a method for calculating the error distance of a combination of 2D joint coordinate values ​​by an AR device when information about the length between hand joints is not stored in the memory, according to an embodiment of the present disclosure.

[0018] Figure 7 This is a diagram illustrating the operation of an AR device according to an embodiment of the present disclosure for calculating the error distance of a combination of 2D joint coordinate values.

[0019] Figure 8 This is a flowchart illustrating a method by which an AR device according to an embodiment of the present disclosure calculates the error distance of image combination based on information about the length between hand joints pre-stored in a memory.

[0020] Figure 9 This is a diagram illustrating the operation of an AR device according to an embodiment of the present disclosure for calculating the error distance of image combination based on information about the length between hand joints pre-stored in memory.

[0021] Figure 10 This is a flowchart illustrating a method for an AR device according to an embodiment of the present disclosure to select image combinations based on error distance and camera priority.

[0022] Figure 11 This is a diagram illustrating the operation of an AR device according to an embodiment of the present disclosure for selecting image combinations based on error distance and camera priority.

[0023] Figure 12 This is a flowchart illustrating the operation of an AR device according to an embodiment of the present disclosure for obtaining 3D position information of a hand joint from at least two second image frames.

[0024] Figure 13 This is a flowchart illustrating a method by which an AR device according to an embodiment of the present disclosure selects an image combination based on an error distance calculated based on estimated 3D joint coordinate values.

[0025] Figure 14This is a diagram illustrating the operation of an AR device according to an embodiment of the present disclosure for obtaining 3D position information of a hand joint based on a combination of 2D joint coordinate values ​​obtained from a combination of some images.

[0026] Figure 15 This is a diagram illustrating the operation of an AR device according to an embodiment of the present disclosure for obtaining 3D position information of a hand joint based on a combination of 2D joint coordinate values ​​obtained from a combination of some images.

[0027] Figure 16 This is a flowchart illustrating a method by which an AR device according to an embodiment of the present disclosure selects a combination of images based on a maximum number of images allowed to be combined.

[0028] Figure 17 This is a diagram illustrating the operation of an AR device according to an embodiment of the present disclosure for obtaining 3D position information of a hand joint based on a combination of 2D joint coordinate values ​​from a combination of some images.

[0029] Figure 18 This is a diagram illustrating the operation of an AR device according to an embodiment of the present disclosure for obtaining 3D position information of a hand joint based on a combination of 2D joint coordinate values ​​obtained from a combination of 2D joint coordinate values ​​obtained from additionally selected image frames. Detailed Implementation

[0030] In accordance with the principles of this disclosure, terms are selected from commonly used terms that are currently widely available. However, these terms may depend on the intent of those skilled in the art, judicial precedent, the emergence of new technologies, etc. Some terms used herein were chosen by the applicant; in such cases, these terms will be described in detail later in conjunction with embodiments of this disclosure. Therefore, throughout this disclosure, these terms should be defined based on their meanings and descriptions.

[0031] As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. All terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0032] The terms “comprising (or including)” or “including (or containing)” are inclusive or open-ended and do not exclude additional, unlisted elements or method steps. As used herein, the terms “unit,” “module,” “block,” etc., all refer to a unit for performing at least one function or operation and may be implemented in hardware, software, or a combination thereof.

[0033] In this disclosure, the expression “configured to” as used herein may be used interchangeably with, for example, “suitable for,” “capable of,” “designed to,” “adapted to,” “made for,” and “capable of.” The expression “configured to” may not necessarily indicate that it is “specifically designed to” in terms of hardware. For example, in some cases, the expression “system configured to do something” may refer to “an entity capable of cooperating with other devices or components to do something.” For example, “processor configured to perform functions A, B, and C” may refer to a dedicated processor (e.g., an embedded processor) for performing functions A, B, and C, or a general-purpose processor (e.g., a central processing unit (CPU) or application processor) that can perform functions A, B, and C by executing one or more software programs stored in memory.

[0034] When the terms “connection” or “coupled” are used, a component may be directly connected to or coupled to another component. However, unless otherwise defined, it should be understood that the component may be indirectly connected to or coupled to that other component via a new component.

[0035] In this disclosure, augmented reality (AR) refers to the display of virtual images or both real objects and virtual images in a real-world physical environment.

[0036] In this disclosure, an AR device is a device capable of presenting AR, which may be, for example, not only glasses-like AR glasses worn on the face by a user, but also head-mounted display devices (HMDs) or AR helmets worn on the head.

[0037] The artificial intelligence (AI) related functions in this disclosure operate via processors and memory. The processors may be configured with one or more processors. These processors may include general-purpose processors (e.g., CPUs, APs, digital signal processors (DSPs), etc.), dedicated graphics processors (e.g., GPUs and vision processing units (VPUs)), or dedicated AI processors (e.g., NPUs). The one or more processors may control the processing of input data according to predefined operating rules or AI models stored in memory. When the one or more processors are dedicated AI processors, the dedicated AI processors may be designed with a hardware architecture specifically for processing a particular AI model.

[0038] The characteristic of predefined operating rules or AI models is that they are built through learning. Specifically, building an AI model through learning refers to constructing predefined operating rules or AI models designed to perform a desired function (or object) when a basic AI model is trained using a learning algorithm with a large amount of training data. This learning can be performed by the device itself that performs AI according to this disclosure, or by a separate server and / or system. Examples of learning algorithms can include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0039] In this disclosure, the AI ​​model can consist of multiple neural network layers. Each of the multiple neural network layers can have multiple weight values, and neural network operations are performed through operations between the results of the previous layer and the multiple weight values. The learning results of the AI ​​model can optimize the multiple weight values ​​included in the multiple neural network layers. For example, multiple weight values ​​can be updated to reduce or minimize the loss or cost values ​​obtained by the AI ​​model during the training process. The artificial neural network model can include, but is not limited to, deep neural networks (DNNs), such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), or deep Q-networks.

[0040] In this disclosure, "visual recognition" refers to image signal processing, which inputs an image into an AI model, performs inference using the AI ​​model, detects objects from the input image, and classifies the objects into a certain category or segments the objects. In embodiments of this disclosure, visual recognition may refer to image processing, which uses an AI model to identify a user's hand from an image captured by a camera and obtains positional information of multiple feature points (e.g., joints) included in the hand.

[0041] In this disclosure, "joint" refers to the part of the human body that connects to the bones, and refers to one or more parts belonging to the hand (e.g., fingers, wrist, palm, etc.) and one or more parts belonging to the upper body (e.g., neck, arm, shoulder, etc.).

[0042] In this disclosure, "inter-joint length" or "inter-joint length of the hand" refers to the length between two joints belonging to the hand.

[0043] Embodiments of this disclosure will now be described in detail with reference to the accompanying drawings to enable those skilled in the art to practice them readily. However, embodiments of this disclosure may be implemented in many different forms and are not limited to the embodiments discussed herein.

[0044] Embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0045] Figure 1This is a conceptual diagram illustrating the operation of an AR device 100 for obtaining three-dimensional (3D) position information of a hand joint, according to embodiments of the present disclosure.

[0046] AR device 100 is a device capable of rendering AR and can be configured, for example, as glasses-like AR glasses to be worn on the face by a user. AR device 100 is shown as... Figure 1 AR glasses, but not limited to. For example, AR device 100 can be implemented as a head-mounted display (HMD) device or AR helmet to be worn on the head.

[0047] refer to Figure 1 The AR device 100 may include multiple cameras 111, 112, 113, and 114, which are the first camera 111 through the fourth camera 114. Figure 1 In the illustrated embodiment, the first camera 111 and the third camera 113 may be located at the upper and lower ends of the frame surrounding the left eye lens of the AR device 100, respectively, and the second camera 112 and the fourth camera 114 may be located at the upper and lower ends of the frame surrounding the right eye lens, respectively. The number and positions of the plurality of cameras 111, 112, 113, and 114 included in the AR device 100 are merely examples and are not limited to this embodiment. Figure 1 The quantity and location are shown. In embodiments of this disclosure, AR device 100 may include at least two (e.g., three, five, six, ..., n) cameras.

[0048] In operation ①, the AR device 100 can obtain multiple images i1, i2, i3 and i4 by using multiple cameras 111, 112, 113 and 14 to photograph the user's hand.

[0049] In step ②, the AR device 100 can obtain 2D joint coordinate values ​​from multiple images i1, i2, i3 and i4.

[0050] In operation ③, the AR device 100 can estimate the 3D joint coordinate values ​​of the hand joint based on a combination of 2D joint coordinate values ​​obtained from an image combination consisting of at least two images.

[0051] In operation ④, the AR device 100 can select the image combination with the smallest error distance from the image combination.

[0052] AR device 100 can obtain 3D position information of hand joints based on a combination of 2D joint coordinate values ​​from at least two images that constitute the selected image combination.

[0053] Now refer to Figure 1 and Figure 2 To describe in detail the functions and / or operations of the AR device 100 for obtaining 3D position information of the hand joints.

[0054] Figure 2 This is a flowchart illustrating an operation method of an AR device 100 according to an embodiment of the present disclosure.

[0055] In operation S210, the AR device 100 obtains 2D joint coordinate values ​​for feature points of the hand joints from multiple images obtained by capturing the user's hand through multiple cameras. See also... Figure 1 In operation ①, the AR device 100 can obtain first images i1 to fourth images i4 by using first cameras 111 to fourth cameras 114 to capture images of the user's hand located in real space. For example, the AR device 100 can obtain first image i1 by using first camera 111 located at the upper left end of the lens frame to capture images of the user's hand, and obtain second image i2 by using second camera 112 located at the upper right end of the lens frame to capture images of the user's hand. The AR device 100 can obtain third image i3 by using third camera 113 located at the lower left end of the lens frame to capture images of the user's hand, and obtain fourth image i4 by using fourth camera 114 located at the lower right end of the lens frame. In this disclosure, for ease of illustration, four images i1 to i4 are shown and described as being obtained by four cameras 111 to 114, but the number of cameras and the number of images are not limited thereto. In embodiments of this disclosure, multiple cameras can be configured with n cameras, where n is 2, 3, 5 or more, and the AR device 100 can obtain n images through n cameras.

[0056] Referring also to operation ②, the AR device 100 can identify feature points of hand joints from each of a plurality of images i1 to i4 and obtain 2D joint coordinate values ​​for those feature points. In this disclosure, a "joint" is a portion that connects to a plurality of bones included in the hand, referring to one or more portions included in a finger, the back of the hand, or the palm. In this disclosure, a feature point can refer to a point in an image that is easily identifiable or distinguishable from the surrounding background. Feature points of hand joints may include, for example, feature points of the wrist joint, feature points of the palm joint, and feature points of the fingers (thumb, index finger, middle finger, ring finger, or little finger).

[0057] In embodiments of this disclosure, AR device 100 can identify feature points of hand joints from a first image i1 to a fourth image i4 through visual recognition using an artificial intelligence (AI) model. The AI ​​model may include a deep neural network (DNN) model trained to identify objects (e.g., a user's hand) and their feature points from image data input from a camera. In embodiments of this disclosure, the DNN model may be a model trained using a supervised learning method that applies tens or hundreds of millions of images as input data and uses the feature points of hand joints included in the input data as ground truth. The DNN model may include at least one of, for example, convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), and deep Q-networks.

[0058] However, this disclosure is not limited to using an AI model to identify feature points of hand joints from the first image i1 to the fourth image i4. In embodiments of this disclosure, the AR device 100 may use known image processing techniques to identify the user's hand from each of the first image i1 to the fourth image i4 and identify feature points for the joints belonging to the hand.

[0059] AR device 100 can obtain 2D joint coordinate values ​​for identified feature points of the hand joints. Figure 1 In the illustrated embodiment, the AR device 100 can obtain the 2D joint coordinate values ​​P of the hand joints identified from the first image i1. 11 To P 1n 2D joint coordinate values ​​P 11 To P 1n This could be the 2D position coordinates of feature points of the hand joints identified from the first image i1. Similarly, the AR device 100 can obtain the 2D joint coordinates P of the hand joints identified from the second image i2. 21 To P 2n Obtain the 2D joint coordinate values ​​P of the hand joints identified from the third image i3. 31 To P 3n And obtain the 2D joint coordinate values ​​P of the hand joints identified from the fourth image i4. 41 To P 4n .

[0060] Return to reference Figure 2 In operation S220, the AR device 100 estimates the 3D joint coordinates of the hand joint based on a combination of 2D joint coordinate values ​​from at least two images composed of a plurality of images. In this disclosure, an image combination can refer to a combination of at least two images. Also referenced is... Figure 1Image combinations may include a first combination {i1, i2, i3, i4} consisting of four images including a first image i1 to a fourth image i4; a second combination {i1, i2, i3} consisting of three images selected from the first image i1 to the fourth image i4; a third combination {i1, i2, i4}; a fourth combination {i1, i3, i4}; a fifth combination {i2, i3, i4}; a sixth combination {i1, i4}; a seventh combination {i1, i3}; an eighth combination {i1, i4}; a ninth combination {i2, i3}; a tenth combination {i2, i4}; and an eleventh combination {i3, i4} consisting of two images selected from the first image i1 to the fourth image i4. In this disclosure, image combinations are shown and described as including a total of eleven combinations, but the number of image combinations is not limited to this. In embodiments of this disclosure, image combinations may include m image combinations, each consisting of at least two images from a total of n images.

[0061] AR device 100 can obtain 3D joint coordinates of a hand joint using triangulation based on a combination of at least two images obtained from two or more of a plurality of cameras 111, 112, 113, and 114, and the positional relationship between the cameras. In embodiments of this disclosure, AR device 100 can estimate 3D joint coordinates of the hand joint using a "ray intersection" method, which obtains 3D positional coordinates based on the intersection point of virtual rays extending from the center positions of the plurality of cameras to the 2D joint coordinate values ​​in at least two images. However, this disclosure is not limited to the foregoing examples, and AR device 100 of this disclosure can use any known method to obtain 3D joint coordinates from a combination of 2D joint coordinate values ​​obtained from a combination of at least two images.

[0062] Also refer to Figure 1 Operation ③, the AR device 100 can base its operation on a combination of 2D joint coordinate values ​​{P} obtained from a combination of the first image i1 and the second image i2. 11 , P 21 To estimate the 3D joint coordinate values ​​P 3D_1 Although not shown in the accompanying drawings, the AR device 100 can estimate 3D joint coordinate values ​​based on a combination of 2D joint coordinate values ​​obtained from at least two component image combinations (first combination to eleventh combination) from each of the first image i1 to the fourth image i4.

[0063] exist Figure 2In operation S230, AR device 100 selects the image combination with the smallest error distance from the image combination, the error distance being calculated based on the estimated 3D joint coordinate values. AR device 100 can measure the inter-joint length of the hand based on the 3D joint coordinate values ​​estimated from the combination of 2D joint coordinate values, and by comparing the measured inter-joint length with the memory 130 (see...). Figure 3 The error distance of the image combination is calculated by comparing the inter-joint lengths stored in the memory 130. In this disclosure, the term "inter-joint length" refers to the length between two joints belonging to the hand. When the memory 130 does not store information about the inter-joint length, the AR device 100 can calculate the error distance resulting from converting 2D joint coordinate values ​​obtained from at least two images constituting each image combination into 3D joint coordinate values. In embodiments of this disclosure, the AR device 100 can obtain information about the center position of a virtual 3D structure formed by rays extending from the center positions of at least two of the plurality of cameras 111, 112, 113, and 114 to the 2D joint coordinate values ​​in at least two images, and calculate the error distance based on the shortest distance from the center position of the virtual 3D structure to each ray.

[0064] When information about the inter-joint length is stored in memory 130, AR device 100 can obtain the inter-joint length by measuring the length between 3D joint coordinate values ​​estimated based on 2D joint coordinate values, and calculate the error distance by comparing the obtained inter-joint length with the inter-joint length stored in memory 130. In embodiments of this disclosure, AR device 100 can calculate the error distance based on the difference between the obtained inter-joint length and the average of the inter-joint lengths pre-stored in memory 130, as well as the standard deviation of the pre-stored inter-joint lengths, and normalize the calculated error distance.

[0065] AR device 100 can select the image combination with the smallest calculated error distance from at least two image combinations. See also: Figure 1 In operation ④, the calculated error distance of the first combination consisting of the first image i1 to the fourth image i4 can be 0.9, and the calculated error distances of the second, third, fourth, and fifth combinations consisting of three of the first image i1 to the fourth image i4 can be 0.8, 0.8, 0.7, and 0.6, respectively. The calculated error distances of the sixth to eleventh combinations consisting of two of the first image i1 to the fourth image i4 can be 0.1, 0.7, 0.2, 0.3, 0.3, and 0.4, respectively. The AR device 100 can select the sixth combination from the first to the eleventh combinations whose error distance is calculated to have the minimum value of 0.1.

[0066] exist Figure 2 In operation S240, the AR device 100 obtains 3D position information of the hand joint based on a combination of 2D joint coordinate values ​​from the selected image set. In embodiments of this disclosure, the AR device 100 may obtain 3D position information of the hand joint based on a combination of 2D joint coordinate values ​​from at least two images constituting the selected image set. (See reference...) Figure 1 In operation ⑤, the AR device 100 can obtain 3D position information of the hand joints based on a combination of 2D joint coordinate values ​​obtained from the first image i1 and the second image i2 of the sixth combination that constitutes the image combination with the minimum error distance, and the positional relationship between the first camera 111 and the second camera 112, using triangulation. However, it is not limited to this, and when the selected image combination includes three or more images, the AR device 100 can obtain 3D position information of the hand joints from the combination of 2D joint coordinate values ​​using a "ray intersection" method.

[0067] For AR device 100 to provide AR services, hand interaction using the user's 3D hand poses and gestures as input is crucial for the input interface. To achieve more realistic AR technology, a technique is needed to obtain 3D positional information of the hand joints, accurately track hand poses (shapes) using this 3D positional information, and recognize gestures. When the accuracy of the 3D positional information of the hand joints is low, the AR device may fail to recognize or misrecognize hand poses or gestures.

[0068] To obtain accurate 3D positional information of hand joints from multiple 2D images acquired by multiple cameras 111, 112, 113, and 114, the accuracy of the 2D positional coordinates of feature points of the hand joints identified from the multiple 2D images is crucial. In vision-based hand tracking technology, improving the accuracy of 3D positional information of hand joints is necessary to enhance the accuracy of hand pose or gesture recognition and provide reliable AR services.

[0069] This disclosure aims to provide an AR device 100 and its operating method for obtaining more accurate 3D position information of hand joints based on a combination of multiple 2D images, in order to improve the accuracy of recognizing the posture or gesture of a user's hand in vision-based hand tracking technology.

[0070] exist Figure 1 and Figure 2In the illustrated embodiment, the AR device 100 improves the accuracy of recognizing a user's hand pose or gesture in vision-based hand tracking technology by: calculating an error distance based on each estimated 3D joint coordinate value from an image combination consisting of multiple images i1, i2, i3, and i4 obtained through multiple cameras 111, 112, 113, and 114; selecting the image combination with the smallest calculated error distance; and obtaining 3D position information of the hand joints based on the 2D joint coordinate values ​​in the selected image combination. As the accuracy of recognizing hand poses or gestures is enhanced, the AR device 100 according to embodiments of this disclosure offers the advantages of providing a stable hand interaction experience for the user and reducing reliance on external controllers or external devices including hand tracking sensors when providing AR services.

[0071] Furthermore, in embodiments of this disclosure, the AR device 100 can obtain 3D position information of the hand joints using only a combination of images obtained from at least two of the multiple cameras 111, 112, 113, and 114. This provides the technical effect of reducing the computational load and saving power required to identify feature points of the hand joints and estimate 3D joint coordinate values ​​for all multiple images. More details will follow later. Figures 10 to 18 The technical effects are described in detail in the embodiments of this disclosure shown.

[0072] Figure 3 This is a block diagram illustrating the components of an AR device 100 according to an embodiment of the present disclosure.

[0073] refer to Figure 3 The AR device 100 may include a camera 110, a processor 120, and a memory 130. The camera 110, processor 120, and memory 130 may be electrically and / or physically connected to each other. Figure 3 Only some components used to describe the operation of AR device 100 are shown in the diagram, but the components included in AR device 100 are not limited to those shown in the diagram. Figure 3 The components are shown. In embodiments of this disclosure, the AR device 100 may further include a communication interface for performing data communication with an external device or server. In embodiments of this disclosure, the AR device 100 may be implemented as a portable device, in which case the AR device 100 may further include a battery for supplying operating power to the camera 110 and the processor 120.

[0074] Camera 110 is configured to acquire images of a hand by capturing images of a hand in real space and the hand itself. Camera 110 may include a lens module, an image sensor, and an image processing module. Camera 110 may acquire still images or videos of an object using an image sensor (e.g., CMOS or CCD). Video may include multiple image frames acquired sequentially by camera 110 capturing the object. The image processing module may encode still images having a single image frame or video data consisting of multiple image frames acquired by the image sensor and send it to processor 120.

[0075] In embodiments of this disclosure, camera 110 may be implemented as a small-sized device to be mounted on portable AR device 100, and may be implemented as a lightweight RGB camera that consumes low power.

[0076] Camera 110 may include two or more cameras. In embodiments of this disclosure, camera 110 may include a first camera 111 (see...). Figure 1 ) to the fourth camera 114 (see Figure 1 For example, when AR device 100 is implemented as glasses-like AR glasses to be worn on a user's face, the first camera 111 and the third camera 113 (see...) Figure 1 The second camera 112 (see above) can be located at the top and bottom of the frame surrounding the left eye lens of the AR device 100, and the second camera 112 (see below) can also be located at the top and bottom of the frame surrounding the left eye lens of the AR device 100. Figure 1 The first camera and the fourth camera 114 can be located at the upper and lower ends of the frame surrounding the right eye lens. Multiple cameras can acquire multiple images, including the user's hand, by photographing it.

[0077] Processor 120 can execute one or more instructions of a program stored in memory 130. Processor 120 may include hardware components for performing arithmetic, logic, input / output operations, and image processing. Figure 3 The element is shown as a component, but is not limited thereto. In embodiments of this disclosure, processor 120 may be configured with one or more components. The one or more processors included in processor 120 may be circuits such as system-on-a-chip (SoC), integrated circuits (ICs), etc.

[0078] Processor 120 may include various processing circuitry and / or multiple processors. For example, the term "processor" as used in this disclosure, including the claims, may include various processing circuitry comprising at least one processor. One or more of the at least one processor may be configured individually and / or collectively in a distributed manner to perform the various functions described in this disclosure. As used herein, a processor, at least one processor, or one or more processors may be configured to perform various functions. However, these terms cover, but are not limited to, situations where one processor performs some functions while other processors perform other functions, and situations where a single processor can perform all functions. Furthermore, at least one processor may include a combination of processors performing the various functions disclosed in a distributed manner. At least one processor may execute program instructions to accomplish or perform various functions.

[0079] Processor 120 may be a general-purpose processor (e.g., a central processing unit (CPU), application processor (AP), digital signal processor (DSP), etc.), a dedicated graphics processor (e.g., a graphics processing unit (GPU), a visual processing unit (VPU), etc.), or a dedicated artificial intelligence (AI) processor (e.g., a neural processing unit (NPU)). Processor 120 may execute at least one instruction or program code stored in memory 150 to control the processing of input data according to predefined operating rules or AI models. In the case where processor 120 is a dedicated AI processor, the dedicated AI processor may be designed with a hardware architecture specifically for processing a particular AI model.

[0080] The memory 130 may include, for example, a storage medium of at least one of the following types: flash memory, hard disk, multimedia card micro-memory, card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), or optical disk.

[0081] Memory 130 may store instructions related to functions and / or operations for obtaining 3D positional information of a hand joint from an image combination consisting of two or more images acquired by AR device 100 via camera 110. In embodiments of this disclosure, memory 130 may store at least one of algorithms, data structures, program code, application programs, and instructions readable by processor 120. The instructions, algorithms, data structures, and program code stored in memory 130 may be implemented, for example, using programming or scripting languages ​​such as C, C++, Java, assembly language, etc.

[0082] The memory 130 may store instructions, algorithms, data structures, or program code relating to the 2D joint coordinate acquisition module 132, the 3D joint coordinate acquisition module 134, and the image combination selection module 136. The modules included in the memory 130 may refer to units that process functions or operations performed by the processor 120, and may be implemented in software such as instructions, algorithms, data structures, or program code. In embodiments of this disclosure, the memory 130 may include an inter-joint length database 138.

[0083] Processor 120 can execute instructions or program code stored in memory 130. The functions and / or operations performed when processor 120 executes instructions or program code of each of the plurality of modules stored in memory 130 will now be described in detail.

[0084] The 2D joint coordinate acquisition module 132 is configured with instructions or program code for performing functions and / or operations to identify feature points of hand joints from an image and obtain 2D joint coordinate values ​​for those feature points. In this disclosure, a "joint" refers to a portion that connects multiple bones included in the hand, specifically one or more portions included in a finger, the back of the hand, or the palm. In this disclosure, a feature point can refer to a point in an image that is easily identifiable or distinguishable from the surrounding background. Feature points of hand joints may include, for example, feature points of the wrist joint, feature points of the palm joint, and feature points of the fingers (thumb, index finger, middle finger, ring finger, or little finger).

[0085] Processor 120 can obtain 2D joint coordinate values ​​for feature points of the hand joints from multiple images acquired by camera 110 by executing instructions or program code of 2D joint coordinate acquisition module 132. In embodiments of this disclosure, multiple cameras can acquire multiple images by photographing the user's hand and provide the image data of the acquired multiple images to processor 120. Processor 120 can identify feature points of the user's hand joints from the multiple images and obtain 2D joint coordinate values ​​for the feature points.

[0086] In embodiments of this disclosure, the 2D joint coordinate acquisition module 132 may include a visual recognition AI model for identifying feature points of hand joints from images. The AI ​​model may include a DNN model trained to identify objects (e.g., a user's hand) from image data and to identify feature points of the objects. In embodiments of this disclosure, the DNN model may be a model trained using a supervised learning method that applies tens of thousands or hundreds of millions of images as input data and applies feature points of hand joints included in the input data as ground truth. The DNN model may include at least one of, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), and a deep Q-network. The processor 120 may input multiple images acquired from multiple cameras into the DNN model and identify feature points of hand joints by inference using the DNN model. The processor 120 may obtain 2D joint coordinate values, which are the 2D position coordinate values ​​of the identified feature points.

[0087] The 3D joint coordinate acquisition module 134 is configured with instructions or program code to perform functions and / or operations for obtaining or estimating 3D joint coordinate values ​​(which are 3D position coordinate values ​​of a hand joint) from a combination of 2D joint coordinate values ​​obtained from at least two images. In embodiments of this disclosure, the 3D joint coordinate acquisition module 134 is configured to obtain or estimate 3D joint coordinate values ​​using triangulation or a ray intersection method. Triangulation uses a combination of 2D joint coordinate values ​​and the positional relationship between multiple cameras used to acquire the images, while ray intersection uses a combination of three or more 2D joint coordinate values ​​and information about the center point between the multiple cameras. The processor 120 can estimate 3D joint coordinate values ​​by executing the instructions or program code of the 3D joint coordinate acquisition module 134, using a combination of 2D joint coordinate values ​​obtained from the image combination. In this disclosure, an image combination can refer to a combination of at least two images. For example, when camera 110 includes n cameras, n images are acquired. In this case, the image combination may include a combination containing n images, a combination containing n-1 images selected from the n images, a combination containing n-2 images selected from the n images, ... or a combination containing nm (which is 2) images selected from the n images.

[0088] The image combination selection module 136 is configured with instructions or program code to perform functions and / or operations for selecting image combinations with the minimum error distance from among the image combinations. The processor 120 can calculate the error distance of each image combination based on estimated 3D joint coordinate values ​​by executing the instructions or program code of the image combination selection module 136, and select the image combination with the minimum calculated error distance from among the image combinations. When the inter-joint length database 138 does not store information about the obtained inter-joint lengths of the hand, the processor 120 can calculate the error distance generated during the conversion of 2D joint coordinate values ​​obtained from at least two images constituting each image combination into 3D joint coordinate values. In embodiments of this disclosure, the processor 120 can obtain information about the center position of a virtual 3D structure formed by rays extending from the center positions of at least two of the plurality of cameras to the 2D joint coordinate values ​​in at least two images, and calculate the error distance based on the shortest distance from the center position of the obtained virtual 3D structure to each ray. The processor 120 can determine the error of the combination of 2D joint coordinate values ​​relative to the sum or average of the calculated error distances. When the inter-joint length database 138 does not store information about the obtained inter-joint length of the hand, it will be combined with Figure 6 and Figure 7 A detailed description of specific embodiments used to calculate error distances is provided.

[0089] When the inter-joint length database 138 in memory 130 stores information about the obtained inter-joint length of the hand, processor 120 can measure the inter-joint length of the hand based on 3D joint coordinate values ​​estimated from at least two images constituting each image combination, and calculate an error distance by comparing the measured inter-joint length with the inter-joint lengths stored in database 138. In embodiments of this disclosure, processor 120 can normalize the error distance based on the difference between the measured inter-joint length and the average of the inter-joint lengths stored in the inter-joint length database 138, and the standard deviation of the stored inter-joint lengths. For example, processor 120 can normalize the error distance for each joint by calculating Mahalanobis distance. In embodiments of this disclosure, processor 120 can determine the error of the image combination based on the sum or average of the error distances calculated for each joint. When the inter-joint length database 138 stores information about the obtained inter-joint length of the hand, it will be combined with... Figure 8 and Figure 9 A detailed description of specific embodiments used to calculate error distances is provided.

[0090] The processor 120 can select the image combination with the smallest calculated error distance for each image combination. In embodiments of this disclosure, when multiple image combinations are identified as having the same or equal to or less than a threshold calculated error distance, the processor 120 can select at least two of the multiple cameras based on a preset priority, and select an image combination consisting of at least two images obtained through the at least two selected cameras. Figure 10 and Figure 11 A detailed description of specific embodiments for selecting image combinations based on camera priority is provided.

[0091] In embodiments of this disclosure, processor 120 can identify image combinations whose calculated error distance exceeds a preset threshold, and skip and not perform error distance calculations for sub-combinations of the identified image combinations. Processor 120 can select the image combination with the smallest error distance from among image combinations whose error distances have been calculated. (This will be combined with...) Figure 13 and Figure 14 A detailed implementation of a specific method for skipping the error distance calculation for sub-combinations of image combinations whose error distance exceeds a threshold is described.

[0092] In embodiments of this disclosure, the processor 120 may set a maximum number of images that can be combined, and calculate the error distance only for image combinations consisting of images whose number is equal to or less than the set maximum number. The processor 120 can select the image combination with the smallest calculated error distance from image combinations consisting of images whose number is equal to or less than the maximum number. Figure 16 and Figure 17 This document describes in detail a specific implementation of selecting an image combination with the minimum error distance when setting the maximum number of images allowed to be combined.

[0093] Processor 120 can obtain 3D position information of the hand joint based on a selected combination of 2D joint coordinate values. In embodiments of this disclosure, processor 120 can obtain 3D position coordinate values ​​of the hand joint based on a selected combination of 2D joint coordinate values ​​and the positional relationship between multiple cameras. For example, processor 120 can use triangulation or a ray intersection method to obtain 3D position coordinate values ​​of the hand joint from a combination of 2D joint coordinate values.

[0094] In embodiments of this disclosure, processor 120 can obtain 3D position information of the hand joint by using 2D joint coordinate values ​​obtained from images other than at least two images constituting an image combination associated with a selected combination of 2D joint coordinate values. Processor 120 can obtain 3D position coordinate values ​​of the hand joint by using at least one 2D joint coordinate value from 2D joint coordinate values ​​obtained from other images not included in the selected image combination that forms a hand joint length similar to the inter-joint length stored in the inter-joint length database 138, and a combination of 2D joint coordinate values ​​obtained from the selected image combination. Figure 18 A specific embodiment is described in detail to obtain the 3D position coordinates of a hand joint by using a combination of 2D joint coordinates obtained from other images and 2D joint coordinates from a selected image combination.

[0095] Although not shown, the AR device 100 may also include a display. The display can show virtual objects that interact with the user's hand. When the AR device is implemented as glasses-like AR glasses, the display can be configured as an optical lens system including a waveguide and an optical engine. The optical engine can be configured with a projector that generates light from virtual objects composed of virtual images and projects the light onto the waveguide. The optical engine may include, for example, an image panel, an illumination optics system, a projection optics system, etc. In embodiments of this disclosure, the optical engine can be arranged on the temples or frame of the AR glasses. In embodiments of this disclosure, the optical engine can display virtual objects by projecting them onto the waveguide under the control of the processor 120.

[0096] However, it is not limited to this, and the display may be configured with at least one of liquid crystal display (LCD), thin film transistor liquid crystal display (TFT-LCD), organic light-emitting diode (OLED), flexible display, 3D display or electrophoretic display.

[0097] Figure 4 This is a flowchart illustrating a method by which an AR device 100 according to an embodiment of the present disclosure obtains 2D joint coordinate values ​​for feature points of a hand joint from multiple images and estimates 3D joint coordinate values ​​of the hand joint based on the obtained 2D joint coordinate values.

[0098] Figure 4 The operations S410 to S430 shown are Figure 2 The detailed operation of operation S210 is shown. Figure 4 The operation S440 shown is Figure 2 The detailed operation of operation S220 is shown. Figure 4 The operation S440 shown can be performed in Figure 2 Before the operation S230 shown.

[0099] In operation S410, the AR device 100 acquires multiple images by capturing images of the user's hand using multiple cameras. In embodiments of this disclosure, the AR device 100 may include multiple cameras, each containing an RGB image sensor. For example, four cameras may be present, but the device is not limited to this. The AR device 100 can acquire multiple 2D images by using multiple cameras to capture images of the user's hand.

[0100] In operation S420, the AR device 100 identifies feature points of the hand joints from each of multiple images and obtains 2D joint coordinate values. In embodiments of this disclosure, the AR device 100 can obtain 2D joint coordinate values ​​for feature points of the hand joints from multiple images by using an AI model for visual recognition. (The above is combined with...) Figures 1 to 3 The visual recognition using an AI model is described, so repeated descriptions will be omitted.

[0101] In operation S430, AR device 100 corrects distortion of 2D joint coordinate values ​​based on distortion correction parameters and the positional relationships between multiple cameras. In this disclosure, distortion model parameters are parameters of a mathematical model used to correct image distortion caused by the physical characteristics of the camera lens. Image distortion models can be defined according to the physical characteristics of the lens. Barrel distortion, Brownian distortion, or pincushion distortion models exist, but are not limited to these. Distortion model parameters may include parameters used to correct images based on a distortion model defined according to the physical characteristics of the lens after images are acquired using the camera. In this disclosure, the positional relationships [R|t] between multiple cameras may include information about the relative positions and orientations of the cameras in a camera layout structure that depends on the size, form, or design of AR device 100. In embodiments of this disclosure, the positional relationships [R|t] between cameras may include a rotation matrix represented by R and a translation vector represented by t. The processor 120 of AR device 100 (see...) Figure 3 The distortion of 2D joint coordinate values ​​can be corrected based on distortion model parameters, which mathematically model the distortion model according to the physical characteristics of the lenses included in each of the multiple cameras and the relative positional relationship [R|t] between the multiple cameras.

[0102] In operation S440, AR device 100 calculates 3D position coordinates of the hand joints based on a combination of 2D joint coordinate values ​​generated from distortion correction and the positional relationships between multiple cameras. In embodiments of this disclosure, processor 120 of AR device 100 corrects the orientation of each of the multiple images based on distortion model parameters and the positional relationships between the multiple cameras. In embodiments of this disclosure, processor 120 may correct the orientation of the multiple images by arranging the epipolar lines of the multiple images in parallel, based on the relative positional relationships [R|t] between the multiple cameras and the distortion model parameters of each of the multiple cameras. Image correction is a technique well known to those skilled in the art, and therefore its detailed description will be omitted.

[0103] In embodiments of this disclosure, processor 120 can calculate 3D joint coordinates of the hand joint using triangulation, which employs corrected 2D joint coordinates derived from distortion correction and adjustment, as well as the positional relationship [R|t] between multiple cameras. However, this is not a limitation, and processor 120 can also calculate 3D joint coordinates of the hand joint using a "ray intersection" method based on the distortion-corrected 2D joint coordinates and the positional relationship [R|t] between multiple cameras.

[0104] Figure 5 This is a flowchart illustrating an AR device 100 according to an embodiment of the present disclosure selecting a combination of 2D joint coordinate values ​​based on an error distance calculated based on estimated 3D joint coordinate values.

[0105] Figure 5 The operations S510 to S550 shown are Figure 2 Detailed operation of S230. Figure 5 The operation shown in S530 or S550 is in Figure 2 E operates before S240.

[0106] In operation S510, AR device 100 determines whether information about inter-joint length is stored in the memory. In embodiments of this disclosure, information about the obtained inter-joint length of the hand may be stored in memory 130 (see [link to documentation]). Figure 3 The inter-joint lengths are stored in the inter-joint length database 138. However, this is not the only possibility, and for example, for a new device or a new user, information about the inter-joint lengths of the hand may not be pre-stored in the inter-joint length database 138.

[0107] When information about the joint length is not stored in memory 130, in operation S520, AR device 100 can calculate the error distance generated when converting 2D joint coordinate values ​​obtained from each image combination consisting of at least two of a plurality of images into 3D joint coordinate values. In embodiments of this disclosure, AR device 100 can obtain information about the center position of a virtual 3D structure formed by rays extending from the center positions of at least two cameras to the 2D joint coordinate values ​​in at least two images, and calculate the error distance based on the shortest distance from the center position of the virtual 3D structure to each ray. Figure 6 and Figure 7 A detailed description of specific embodiments of operation S520 is provided.

[0108] In operation S530, AR device 100 selects the image combination with the smallest calculated error distance from the image combinations. AR device 100 can compare the error distances calculated for each combination of 2D joint coordinate values ​​obtained from each image combination and select the image combination with the smallest error distance.

[0109] In operation S540, the AR device 100 obtains 3D joint coordinate values ​​based on a combination of 2D joint coordinate values ​​from the selected image set. In embodiments of this disclosure, the AR device 100 may calculate 3D joint coordinate values ​​based on 2D joint coordinate values ​​obtained from at least two images constituting the selected image set and the positional relationship between at least two cameras configured to obtain at least two images. (The above is in conjunction with...) Figure 4 The specific method for calculating 3D joint coordinate values ​​by AR device 100 is described, so repeated descriptions will be omitted.

[0110] When information about interarticular length is stored in memory 130 (see...) Figure 3 The inter-joint length database 138 (see) Figure 3In operation S550, the AR device 100 calculates the error distance by comparing the inter-joint length of the hand measured based on the estimated 3D joint coordinate values ​​with the inter-joint lengths stored in the memory. In embodiments of this disclosure, the AR device 100 may measure the inter-joint length of the hand based on 3D joint coordinate values ​​estimated from a combination of 2D joint coordinate values, and calculate the error distance by comparing the measured inter-joint length with the average of the inter-joint lengths stored in the inter-joint length database 138. In embodiments of this disclosure, the AR device 100 may calculate the error distance for each joint included in the hand, and determine the error distance of the combination of 2D joint coordinate values ​​by adding the error distances calculated for each joint. However, this is not a limitation, and in embodiments of this disclosure, the AR device 100 may determine the error distance of the combination of 2D joint coordinate values ​​by averaging the error distances calculated for each joint included in the hand.

[0111] In operation S560, AR device 100 selects the image combination with the smallest calculated error distance from the image combinations. In embodiments of this disclosure, AR device 100 may compare the error distances calculated for each combination of 2D joint coordinate values ​​obtained from each image combination and select the image combination with the smallest error distance.

[0112] In operation S570, AR device 100 obtains 3D joint coordinate values ​​based on a combination of 2D joint coordinate values ​​from the selected image combination. In embodiments of this disclosure, AR device 100 may calculate 3D joint coordinate values ​​based on 2D joint coordinate values ​​obtained from at least two images constituting the selected image combination and the positional relationship between at least two cameras configured to obtain images for obtaining 2D joint coordinate values.

[0113] In operation S580, AR device 100 stores the inter-joint lengths measured based on the generated 3D joint coordinate values. In embodiments of this disclosure, AR device 100 can measure the inter-joint lengths included in the hand based on the generated 3D joint coordinate values ​​and store the measured inter-joint lengths in inter-joint length database 138 (see [link to documentation]). Figure 3 )middle.

[0114] Figure 6 This illustrates an embodiment of the present disclosure when memory 130 (see...) Figure 3 A flowchart illustrating the method by which an AR device 100 calculates the error distance of a combination of 2D joint coordinate values ​​when no information about the length between hand joints is stored in the device.

[0115] Figure 6 The operations S610 and S620 shown are Figure 5 Detailed operation of the S520. (During execution...) Figure 5 After operating the S510, you can execute... Figure 6 The operation S610 is shown. Figure 6 The operation shown in S620 can be performed in Figure 5 Before operating the S530.

[0116] Figure 7 This is a diagram illustrating the operation of an AR device 100 according to an embodiment of the present disclosure for calculating error distances ED_1 and ED_2 of a combination {P1, P2} of 2D joint coordinate values ​​from image combinations {i1, i2}.

[0117] Now refer to Figure 6 and Figure 7 Describe the functions and / or operations of AR device 100 for calculating the error distances ED_1 and ED_2 of the combination {P1, P2} of 2D joint coordinate values ​​when information about the length between hand joints is not stored in memory 130.

[0118] exist Figure 6 In operation S610, the AR device 100 obtains information about the center position of a virtual 3D structure formed by rays extending from the center positions of at least two cameras to the 2D joint coordinate values ​​in at least two images. Also refer to... Figure 7 In the illustrated embodiment, the processor 120 of the AR device 100 (see...) Figure 3 A virtual 3D structure 700 can be formed based on a first ray 710 extending from the center position of the first camera 111 toward the first 2D joint coordinate value P1 in the first image i1 and a second ray 720 extending from the center position of the second camera 112 toward the second 2D joint coordinate value P2 in the second image i2. The virtual 3D structure 700 can be in the form of a cuboid, comprising coordinates A1 and D1 formed on the first ray 710 and coordinates C2 and D2 formed on the second ray 720 as vertices. However, it is not limited to this, and the virtual 3D structure 700 can be formed in the shape of a polyhedron, cylinder, or sphere. The processor 120 can obtain 3D position information of the center C of the virtual 3D structure 700. The processor 120 can obtain the 3D position information of the center C based on, for example, information about the center of gravity or centroid of the virtual 3D structure 700.

[0119] refer to Figure 6 In operation S620, AR device 100 calculates the error distance based on the shortest distance from the center of the virtual 3D structure to each ray. See also... Figure 7The processor 120 of the AR device 100 can calculate a first error distance ED_1 based on the shortest distance between the center C of the virtual 3D structure 700 and the first ray 710. The first error distance ED_1 can also be calculated based on the distance from the end of a vertical line starting from the center C of the virtual 3D structure 700 to the first ray 710. Similarly, the processor 120 can calculate a second error distance ED_2 based on the shortest distance between the center C of the virtual 3D structure 700 and the second ray 720.

[0120] The processor 120 can determine the error distance of the combination {P1, P2} of 2D joint coordinate values ​​by calculating the sum or average of the first error distance ED_1 and the second error distance ED_2. Figure 7 For ease of illustration, an embodiment is shown for calculating the error distance of a combination {P1, P2} of 2D joint coordinate values ​​obtained from a first image i1 and a second image i2 obtained using a first camera 111 and a second camera 112, but this disclosure is not limited thereto. Figure 7 In the illustrated method, processor 120 can calculate the error distance of a combination of 2D joint coordinate values ​​obtained from a combination of at least two images. Furthermore, in embodiments of this disclosure, AR device 100 can calculate the error distance of a combination of 2D joint coordinate values ​​using any known technique for calculating the error distance generated during the conversion of the combination of 2D joint coordinate values ​​to 3D joint coordinate values.

[0121] Figure 8 This illustrates an AR device 100 according to an embodiment of the present disclosure, using a pre-stored memory 130 (see [link]). Figure 3 The flowchart shows a method for calculating the error distance of image combination using information about the length between hand joints.

[0122] Figure 8 The operations S810 and S820 shown are Figure 5 Detailed operation of the S550. (During execution...) Figure 5 After operating the S510, you can execute... Figure 8 The operation S810 is shown. Figure 8 The operation shown in S820 can be performed in Figure 5 Before operating the S560.

[0123] Figure 9 An AR device 100 according to an embodiment of the present disclosure is shown for use with pre-stored memory 130 (see [link]). Figure 3 The diagram illustrates the operation of calculating the error distance of image combination using information about the length between hand joints.

[0124] Now refer to Figure 8 and Figure 9Describe the function and / or operation of AR device 100 for calculating the error distance of a combination of 2D joint coordinate values ​​obtained from image combination when information about the obtained inter-joint length is stored in inter-joint length database 138.

[0125] exist Figure 8 In operation S810, the AR device 100 measures the length between hand joints based on estimated 3D joint coordinate values. The AR device 100 can measure the length between the joints included in the hand by using 3D joint coordinate values ​​estimated based on a combination of 2D joint coordinate values. See also... Figure 9 The processor 120 of AR device 100 (see Figure 3 The 3D joint coordinate value P can be obtained based on the combination of 2D joint coordinate values ​​from at least two images from each of the multiple images i1 to i4. 3D_1 P 3D_2 P 3D_3 ... and P 3D_n The processor 120 can base its calculations on the obtained 3D joint coordinate values ​​P. 3D_1 P 3D_2 P 3D_3 ... P 3D_n To measure the length of the joints in the hand i .exist Figure 9 In the illustrated embodiment, the processor 120 can measure a first 3D joint coordinate value P representing the wrist-metacarpal joint. 3D_1 And represents the coordinate value P of the second 3D joint of the hand, which includes the metacarpophalangeal joints. 3D_2 The length l between i ( Figure 9 (Length of the metacarpals in the embodiment).

[0126] Return to reference Figure 8 In operation S820, the AR device 100 calculates the error distance based on the measured hand joint length and joint length information stored in memory. See also... Figure 9 The processor 120 of the AR device 100 can process the measured inter-joint length l i Compared with the inter-joint length l pre-stored in the inter-joint length database 138 Ri The error distance is calculated by comparison. In embodiments of this disclosure, processor 120 may accumulate information about the inter-joint length measured from each of n image frames and store the accumulated information in an inter-joint length database 138, and estimate the accumulated information about the inter-joint length l using a Gaussian distribution. Ri The processor 120 can use the estimated mean and standard deviation of the Gaussian distribution to calculate the newly measured interarticular length l of the hand. iThe Mahalanobis distance is calculated and then defined for the interarticular length l. i The error distance.

[0127] [Equation 1]

[0128] Mahalanobis distance:

[0129] Referring to Equation 1, processor 120 can calculate the Mahalanobis distance d by performing the following operations. i : The inter-joint length x calculated based on 3D joint coordinates i Compared with the inter-joint length l stored in the inter-joint length database 138 Ri The average value μ i The square of the difference between them divided by the stored inter-joint length l Ri The standard deviation σ is the square of the distance. Mahalanobis distance is calculated based on the measured interarticular length l. i and the inter-joint length l stored in the inter-joint length database 138 Ri The difference between them is the value obtained by normalizing the error distance, and the processor 120 can calculate the Mahalanobis distance d. i The error distance for the i-th joint is determined. In embodiments of this disclosure, the processor 120 can calculate the Mahalanobis distance d for all joints included in the hand. i And through the calculated Mahalanobis distance d i The sum of these values ​​is used to calculate the error distance of the image combination. However, this is not the only limitation, and in embodiments of this disclosure, the processor 120 can use the Mahalanobis distance d of each joint. i The average or weighted sum is used to calculate the error distance of the image combination.

[0130] Figure 10 This is a flowchart illustrating a method by which an AR device 100, according to an embodiment of the present disclosure, selects an image combination based on error distance and camera priority.

[0131] Figure 10 The operations S1010 to S1030 shown are Figure 2 Detailed operation of S230. During execution... Figure 2 After operation S220, it can be executed. Figure 10 The operation shown is S1010. Figure 10 The operation S1030 shown can be performed in Figure 2 Before operation S240.

[0132] Figure 11 This is a diagram illustrating the operation of an AR device 100 according to an embodiment of the present disclosure for selecting image combinations based on error distance and the priority of cameras 111 to 114.

[0133] Now refer to Figure 10 and Figure 11 The operation of AR device 100 for selecting a combination of 2D joint coordinate values ​​based on error distance and the priority of cameras 111 to 114 is described.

[0134] exist Figure 10 In operation S1010, the AR device 100 identifies whether there are multiple image combinations with the same error distance or with an error distance equal to or less than a threshold. In embodiments of this disclosure, the AR device 100 can identify whether there are two or more combinations of 2D joint coordinate values ​​with the same error distance, or whether there are combinations of 2D joint coordinate values ​​with an error distance equal to or less than a preset threshold, wherein the error distance is calculated for each image combination consisting of two or more images.

[0135] Also refer to Figure 11 AR device 100 can obtain a first image i1 by photographing the user's hand using a first camera 111. Similarly, AR device 100 can obtain a second image i2 using a second camera 112, a third image i3 using a third camera 113, and a fourth image i4 using a fourth camera 114. AR device 100 can obtain a combination of 2D joint coordinate values ​​from each of at least two of the image combinations (first combination to eleventh combination) consisting of the first image i1 to the fourth image i4. Figure 11 In the illustrated embodiment, within the image combinations, the first combination may include all first images i1 to fourth images i4, the second to fifth combinations may be image combinations including three images selected from first images i1 to fourth images i4, and the sixth to eleventh combinations may be image combinations including two images selected from first images i1 to fourth images i4. For ease of illustration, the number of multiple images i1 to i4 and the number of image combinations are used as examples, and the number of multiple images and the number of image combinations are not limited to these examples. Figure 11 The quantity shown.

[0136] The processor 120 of the AR device 100 (see...) Figure 3 The processor 120 can estimate 3D joint coordinate values ​​based on a combination of 2D joint coordinate values ​​obtained from each of the image combinations (first to eleventh combinations), and calculate the error distance for each image combination based on the estimated 3D joint coordinate values. The method by which the processor 120 estimates 3D joint coordinate values ​​based on the 2D joint coordinate values ​​of feature points of the hand joint obtained from the image combinations and calculates the error distance based on the estimated 3D joint coordinate values ​​is similar to... Figures 1 to 9 The methods described are the same, so repeated descriptions will be omitted.

[0137] AR device 100 can identify whether there are multiple image combinations with the same or equal to or less than a preset threshold error distance. The error distance is calculated for each image combination (the first to the eleventh combination). Figure 11 In the illustrated embodiment, the second and third combinations have the same error distance of 0.8, the sixth and eighth combinations have the same error distance of 0.2, and the ninth and tenth combinations have the same error distance of 0.3. Furthermore, when the error distance threshold is preset to 0.3, the image combinations with an error distance equal to or less than the threshold are the sixth, eighth, ninth, and tenth combinations. The processor 120 of the AR device 100 can identify the second and third combinations, the sixth and eighth combinations, and the ninth and tenth combinations with the same error distance among the image combinations including the first to eleventh combinations, and identify the sixth, eighth, ninth, and tenth combinations with an error distance equal to or less than the threshold.

[0138] Return to reference Figure 10 In operation S1020, the AR device 100 selects at least two of a plurality of cameras based on a preset priority. See also... Figure 11 The priorities of multiple cameras 111 to 114 can be preset. For example, among the multiple cameras 111 to 114, the first camera 111 can be set as the first priority, the fourth camera 114 as the second priority, the second camera 112 as the third priority, and the third camera 113 as the fourth priority. However, it is not limited to this, and in the embodiments of this disclosure, the AR device 100 can receive user input (e.g., finger pointing input) to determine the priorities of the multiple cameras 111 to 114, and determine the priority of each of the multiple cameras 111 to 114 based on the received user input.

[0139] The processor 120 of the AR device 100 can select at least two of a plurality of cameras 111 to 114 based on priority. For example, the processor 120 can select a first camera 111 and a fourth camera 114 from the plurality of cameras 111 to 114 based on a set priority.

[0140] Return to reference Figure 10 In operation S1030, the AR device 100 selects an image combination consisting of at least two images captured and obtained by at least two selected cameras. In embodiments of this disclosure, the AR device 100 can select an image combination consisting of images obtained by at least two cameras selected based on a set priority from a plurality of image combinations having the same error distance or an error distance equal to or less than a threshold. Also refer to... Figure 11In the embodiment shown, the processor 120 of the AR device 100 can select from the sixth, eighth, ninth, and tenth combinations that have the same error distance or an error distance equal to or less than a threshold, the eighth combination consisting of the first image i1 and the fourth image i4 obtained by the first camera 111 and the fourth camera 114 selected based on a set priority.

[0141] Processor 120 can base its work on a combination of 2D joint coordinate values ​​{P} obtained from the first image i1 and the fourth image i4 constituting the eighth combination. 11 , P 41 To obtain the 3D position information (3D position coordinates P) of the hand joints. 3D ).

[0142] As the accuracy of 2D joint coordinate values ​​identified from images acquired via a camera improves, a higher priority can be assigned to the camera. Figure 10 and Figure 11 In the illustrated embodiment, when multiple image combinations with the same error distance or an error distance equal to or less than a threshold are identified, the AR device 100 can select at least two cameras based on a set priority, and obtain 3D position information of the hand joint from at least two images obtained by using at least two selected cameras, thereby providing a technical effect of improving the accuracy of the 3D position information of the hand joint.

[0143] Figure 12 This is a flowchart illustrating the operation of an AR device 100 according to an embodiment of the present disclosure for obtaining 3D position information of a hand joint from at least two second image frames.

[0144] In operation S1210, the AR device 100 obtains 3D position information of the hand joints from the image combination of the first image frame. In embodiments of this disclosure, the processor 120 of the AR device 100 (see...) Figure 3 The processor 120 can select an image combination of first image frames with the smallest error distance from a plurality of first image frames obtained at a first time point, and obtain 3D position information of the hand joint based on a combination of 2D joint coordinate values ​​from a combination of at least two first image frames constituting the selected image combination. For example, the processor 120 can use triangulation or a "ray intersection" method to obtain 3D position information of the hand joint based on a combination of 2D joint coordinate values ​​from the image combination of the first image frames.

[0145] In operation S1220, AR device 100 selects at least two cameras from a plurality of cameras that are configured to acquire at least two images, and the selected image combination is acquired from the at least two images. For example, when the image combination selected in operation S1210 is a combination of a first image, a second image, and a fourth image, processor 120 can select a first camera, a second camera, and a fourth camera from a plurality of cameras that are respectively configured to acquire the first image, the second image, and the fourth image.

[0146] In operation S1230, AR device 100 can use at least two selected cameras to obtain at least two second image frames. For example, in operation S1220, when a first camera, a second camera, and a fourth camera are selected, processor 120 can obtain second image frames (including the first image, the second image, and the fourth image) by using only the first camera, the second camera, and the fourth camera to photograph the user's hand at a second time point.

[0147] In operation S1240, the AR device 100 obtains 3D position information of the hand joint based on a combination of 2D joint coordinate values ​​obtained from a combination of at least two second image frames. For example, in operation S1230, when second image frames (including a first image, a second image, and a fourth image) are obtained using a first camera, a second camera, and a fourth camera, the processor 120 can obtain 3D position information of the hand joint, i.e., 3D joint coordinate values, based on a combination of 2D joint coordinate values ​​obtained from a combination of second image frames including the first image, the second image, and the fourth image.

[0148] according to Figure 12 The AR device 100 of the illustrated embodiment selects at least two cameras configured to acquire at least two images from a combination of images of first image frames acquired at a previous time point (first time point) that have the minimum error distance. The AR device 100 also acquires 3D position information of the hand joint from a combination of at least two second image frames acquired at a later time point (second time point) using the at least two selected cameras. This eliminates the need for calculations such as identifying hand joint feature points, obtaining 2D joint coordinate values, and estimating 3D joint coordinate values, thus providing the technical effects of reducing computational load and saving power.

[0149] Figure 13 This is a flowchart illustrating a method by which an AR device 100 according to an embodiment of the present disclosure selects an image combination based on an error distance calculated based on estimated 3D joint coordinate values.

[0150] Figure 13 The operations S1310 to S1330 shown are Figure 2 Detailed operation of S230. During execution... Figure 2After operation S220, it can be executed. Figure 13 The operation S1310 is shown. Figure 13 The operation S1330 shown can be performed in Figure 2 Before the operation S240 shown.

[0151] Figure 14 This illustrates an AR device 100 according to an embodiment of the present disclosure for combining 2D joint coordinate values ​​obtained from some of the image combinations {P}. 11 , P 21 To obtain the 3D position information of the hand joints P 3D The diagram shows the operation.

[0152] Now refer to Figure 13 and Figure 14 Describe the operation of AR device 100.

[0153] refer to Figure 13 In operation S1310, the AR device 100 identifies image combinations whose calculated error distance exceeds a preset threshold. In embodiments of this disclosure, the preset threshold is an upper limit on the error distance at which the accuracy of the 3D positional information to be calculated may fall below a reference value, and can be preset. For example, the preset threshold can be 1. However, it is not limited thereto. Also refer to... Figure 14 In the illustrated embodiment, the AR device 100 can obtain first images i1 to fourth images i4 by photographing the user's hand using first cameras 111 to fourth cameras 114. It estimates 3D joint coordinate values ​​based on combinations of 2D joint coordinate values ​​obtained from each of at least two of the image combinations (first combination to eleventh combination), and calculates the error distance for each image combination based on the estimated 3D joint coordinate values. Figure 14 In the illustrated embodiment, the description of the image combination, apart from the error distance, is similar to... Figure 11 The embodiments are the same, and repeated descriptions will be omitted. See references. Figure 14 In the illustrated embodiment, the processor 120 of the AR device 100 (see...) Figure 3 It can identify a fifth combination from among image combinations including the first to the eleventh combinations, where the error distance is 1.7 and exceeds a threshold (e.g., 1).

[0154] Return to reference Figure 13 In operation S1320, the AR device 100 skips the calculation of the error distance for sub-combinations of the recognized image combinations instead of performing the calculation. See also... Figure 14In the illustrated embodiment, the processor 120 may not calculate the error distance of the sub-combinations of the second image i2, the third image i3, and the fourth image i4 constituting the fifth combination. For example, the processor 120 may skip the calculation of the error distance for the ninth combination of the second image i2 and the third image i3, the tenth combination of the second image i2 and the fourth image i4, and the eleventh combination of the third image i3 and the fourth image i4.

[0155] refer to Figure 13 In operation S1330, the AR device 100 selects the image combination with the smallest error distance from the image combinations with calculated error distances. See also... Figure 14 In the illustrated embodiment, the processor 120 of the AR device 100 can select a sixth combination with the smallest error distance from the first to eleventh combinations, excluding the ninth, tenth, and eleventh combinations. The processor 120 can select this sixth combination based on a combination {P} of 2D joint coordinate values ​​obtained from the first image i1 and the second image i2 constituting the sixth combination. 11 , P 21 To obtain the 3D position information of the hand joints P 3D .

[0156] exist Figure 13 and Figure 14 In the illustrated embodiment, AR device 100 can search for and obtain 3D position information P of the hand joint. 3D Among the image combinations (first to eleventh combinations), the combination whose error distance exceeds the preset reference value relative to accuracy is identified. Figure 14 The fifth combination in the illustrated embodiment is used, and the calculation of the error distance for the sub-combinations of the identified combination is skipped, thereby improving the search speed. Furthermore, in embodiments of this disclosure, the AR device 100 skips the calculation of the error distance for the sub-image combinations, thereby providing the technical effect of reducing computational load and thus saving power.

[0157] Figure 15 This illustrates an AR device 100 according to an embodiment of the present disclosure for combining 2D joint coordinate values ​​obtained from some of the image combinations {P}. 11 , P 31 To obtain the 3D position information of the hand joints P 3D The diagram shows the operation.

[0158] refer to Figure 15The AR device 100 can obtain first images i1 to fourth images i4 by capturing images of the user's hand using first cameras 111 to fourth cameras 114. Generally, when images are acquired using an ultra-wide-angle lens such as a fisheye lens, the image has relatively small distortion errors in the central region and relatively large distortion errors in the edge regions. Referring to distortion error image 1500, the central region 1510 of the image has small distortion errors, while the edge regions 1520 of the entire image region, excluding the central region 1510, have relatively large distortion errors. Figure 15 In the illustrated embodiment, the user's hand is located in the central region of the first image i1 and the third image i3, and in the edge region of the fourth image i4. In the case of the second image i2, the user's hand is outside the field of view of the second camera 112, therefore the user's hand is not included in the second image i2.

[0159] The processor 120 of the AR device 100 (see...) Figure 3 It can identify the user's hand from each of multiple images i1 to i4, and identify at least one image in which the identified hand is located on an edge with a relatively high level of distortion within the entire image area. Figure 15 In the illustrated embodiment, processor 120 can identify a user's hand from each of a plurality of images i1 to i4, and identify a fourth image i4 in which the identified hand is located at the edge of the entire image area. In embodiments of this disclosure, processor 120 can identify a second image i2 in which the user's hand is not identified among the plurality of images i1 to i4.

[0160] When combining images, processor 120 can exclude images with large distortion areas. Figure 15 In the illustrated embodiment, the processor 120 can combine images, but does not need to anticipate a fourth image i4 with significant distortion due to the hand being located at the edge of the entire image area, nor a second image i2 where the hand is not recognized. By excluding the second image i2 and the fourth image i4, the processor 120 can obtain an image combination including the first image i1 and the third image i3.

[0161] Processor 120 can base its work on a combination of 2D joint coordinate values ​​{P} obtained from a combination of images including first image i1 and third image i3. 11 , P 31 This is used to estimate 3D joint coordinate values, and the error distance is calculated based on the estimated 3D joint coordinate values. For example, the error distance can be calculated as 0.7.

[0162] Processor 120 can base its 2D joint coordinate values ​​{P} on the image combination obtained from the first image i1 and the third image i3. 11, P 31 To obtain the 3D position information of the hand joints P 3D .

[0163] exist Figure 15 In the illustrated embodiment, in order to combine at least two of the multiple images i1 to i4, the AR device 100 can exclude images expected to have relatively large distortion errors from the image combination (in... Figure 15 In the embodiments described, the second image i2 and the fourth image i4 are used, and an image combination containing only the other images is obtained. Therefore, according to embodiments of this disclosure, the AR device 100 improves the accuracy of 3D position information of hand joints. Furthermore, in embodiments of this disclosure, the AR device 100 provides the technical effect of reducing the computational load used to calculate error distances by reducing the number of image combinations, thereby saving power consumption.

[0164] Figure 16 This is a flowchart illustrating a method by which an AR device 100 according to an embodiment of the present disclosure selects some of the images in a combination based on a maximum number of images that can be combined.

[0165] Figure 16 Operations S1610 to S1630 shown are Figure 2 Detailed operation of S230. During execution... Figure 2 After operating S220, you can execute... Figure 16 The operation shown is S1610. Figure 16 The operation S1630 shown can be performed in Figure 2 Before the operation S240 shown.

[0166] Figure 17 This illustrates an AR device 100 according to an embodiment of the present disclosure for combining 2D joint coordinate values ​​from some of the image combinations {P}. 11 , P 21 To obtain the 3D position information of the hand joints P 3D The diagram shows the operation.

[0167] Now refer to Figure 16 and Figure 17 Describes the AR device 100 for selecting some from a combination of images and based on a combination of 2D joint coordinate values ​​obtained from the selected image combination {P}. 11 , P 21 To obtain the 3D position information of the hand joints P 3D The operation.

[0168] exist Figure 16In operation S1610, the AR device 100 sets the maximum number of images that can be combined from a plurality of images. In embodiments of this disclosure, the maximum number of images that can be combined is the maximum number of images that constitute a combination from the plurality of images, which can be less than or equal to the total number of images. For example, when the number of images is 4, the maximum number of images that can be combined can have a value of 4 or less (e.g., 3). In embodiments of this disclosure, the maximum number of images that can be combined can be preset. However, this is not a limitation, and in embodiments of this disclosure, the AR device 100 can receive user input for inputting the maximum number of images that can be combined, and determine the maximum number of images that can be combined based on the received user input.

[0169] In operation S1620, the AR device 100 calculates the error distance only for image combinations consisting of images whose number is equal to or less than the set maximum number. (Reference) Figure 17 The AR device 100 can obtain first images i1 to fourth images i4 by photographing the user's hand using first cameras 111 to fourth cameras 114. Figure 17 In the illustrated embodiment, the maximum number of images that can be combined is three. The processor 120 of the AR device 100 (see...) Figure 3 The processor 120 can obtain image combinations consisting of at least two and at most three images from the first image i1 to the fourth image i4 (the second to eleventh combinations). The second to fifth combinations are image combinations consisting of three images from the first image i1 to the fourth image i4, and the sixth to eleventh combinations are image combinations consisting of two images from the first image i1 to the fourth image i4. The processor 120 can estimate 3D joint coordinate values ​​based on the combination of 2D joint coordinate values ​​obtained from each of the second to eleventh combinations, and calculate the error distance for each image combination based on the estimated 3D joint coordinate values.

[0170] Return to reference Figure 16 In operation S1630, the AR device 100 selects the image combination with the smallest calculated error distance. (Reference) Figure 17 The processor 120 of the AR device 100 can select a sixth combination from a second to an eleventh combination composed of a maximum number (e.g., 3) of images, where the calculated error distance is the minimum value of 0.1. The processor 120 can select this sixth combination based on a combination of 2D joint coordinate values ​​{P} obtained from the first image i1 and the second image i2 constituting the sixth combination. 11 , P 21 To obtain the 3D position information of the hand joints P 3D .

[0171] exist Figure 16 and Figure 17 In the illustrated embodiment, the AR device 100 sets a maximum number of images that can constitute an image combination, and selects the image combination with the smallest error distance from the image combinations composed of images whose individual numbers are equal to or less than the set maximum number, thereby providing the technical effect of reducing the amount of computation used to calculate the error distance and saving power consumption.

[0172] Figure 18 This illustrates an AR device 100 according to an embodiment of the present disclosure for using 2D joint coordinate values ​​{P} derived from image combinations {i1,i2}. 11 , P 21} and the 2D joint coordinate values ​​P obtained from the additional selected image frame i4 41 The combination of these components is used to obtain the 3D position information of the hand joints. 3D The diagram shows the operation.

[0173] refer to Figure 18 The processor 120 of AR device 100 (see Figure 3 The sixth combination, which has the smallest error distance, can be selected from the first to the eleventh combinations (each being a combination of at least two images from the first image i1 to the fourth image i4). This excludes the combination of 2D joint coordinate values ​​obtained from the first image i1 and the second image i2 constituting the sixth combination {P}. 11 ,P 21 In addition, processor 120 can also select 2D joint coordinate values ​​P 41 In embodiments of this disclosure, the processor 120 can measure the inter-joint length l based on 3D joint coordinate values ​​obtained from images i3 and i4, other than the first image i1 and the second image i2 constituting the sixth combination, from a plurality of images i1 to i4. i And select the measurement during which the data is stored in memory 130 (see...) Figure 3 The inter-joint lengths in the same joint lengths are l. i The 2D joint coordinate values. In embodiments of this disclosure, the processor 120 may additionally select the 2D joint coordinate values ​​P obtained from the fourth image i4. 41 .

[0174] In embodiments of this disclosure, processor 120 may select 2D joint coordinate values ​​that satisfy kinematics from at least two joint coordinate values ​​obtained from other images i3 and i4. In this disclosure, the term "kinematics" may refer to the range of motion of a joint constrained by the anatomy of the human musculoskeletal system. Inconsistencies in kinematics include, for example: i) incorrect measurement of the wrist joint such that the length from the back of the hand to the wrist is longer than the fingers, or a particular finger is too long; ii) finger joints flexing outwards beyond their range of motion; and iii) a first interphalangeal joint of a finger flexing outwards beyond its range of motion and a second interphalangeal joint flexing inwards. Figure 18 In the illustrated embodiment, the processor 120 can select 2D joint coordinate values ​​(e.g., P) representing feature points of a hand joint within a range of joint motion that satisfies the anatomical constraints of the human musculoskeletal system. 41 ).

[0175] The processor 120 of the AR device 100 can be based on not only 2D joint coordinate values ​​obtained from the selected image combination (sixth combination) but also additionally selected 2D joint coordinate values ​​P. 41 The combination of these methods is used to obtain the 3D position information of the hand joints. 3D .

[0176] according to Figure 18 The AR device 100 of the illustrated embodiment improves the accuracy of 3D position information by using not only 2D joint coordinate values ​​from the selected image combination, but also additional 2D joint coordinate values ​​selected based on the length between hand joints.

[0177] This disclosure provides a method for an AR device 100 to obtain 3D position information of a hand joint. According to embodiments of this disclosure, an operation method of an AR device may include: in operation S210, obtaining 2D joint coordinate values ​​for feature points of the hand joint from multiple images obtained by capturing a user's hand through multiple cameras 111, 112, 113, and 114. The operation method of the AR device 100 may include: in operation S220, obtaining 3D joint coordinate values ​​of the hand joint based on a combination of 2D joint coordinate values ​​obtained from at least two images constituting a plurality of images. The operation method of the AR device 100 may include: in operation S230, selecting an image combination with the smallest error distance from the image combination, the error distance being calculated based on the obtained 3D joint coordinate values. The operation method of the AR device 100 may include: in operation S240, obtaining 3D position information of the hand joint based on a combination of 2D joint coordinate values ​​from at least two images constituting the selected image combination.

[0178] In embodiments of this disclosure, selecting an image combination in operation S230 may include: when the length between hand joints is not stored in the memory, in operation S520, calculating the error distance generated when converting 2D joint coordinate values ​​obtained from at least two images constituting each image combination into 3D joint coordinate values; and in operation S530, selecting the image combination with the smallest calculated error distance from the image combinations.

[0179] In embodiments of this disclosure, calculating the error distance in operation S520 may include: in operation S610, obtaining information about the center position of a virtual 3D structure formed by rays extending from the center positions of at least two of the plurality of cameras 111, 112, 113 and 114 to 2D joint coordinate values ​​in at least two images; and in operation S620, calculating the error distance based on the shortest distance from the center position of the virtual 3D structure to each ray.

[0180] In embodiments of this disclosure, selecting an image combination in operation S230 may include: in operation S810, measuring the inter-joint length of the hand based on the obtained 3D joint coordinate values; and in operation S820, calculating an error distance based on the measured inter-joint length and information about the inter-joint length stored in memory. Selecting an image combination in operation S230 may also include: in operation S560, selecting the image combination with the smallest calculated error distance from the image combinations.

[0181] In embodiments of this disclosure, calculating the error distance in operation S820 may include: normalizing the error distance based on the difference between the stored mean of inter-joint lengths and the measured inter-joint lengths of the hand, as well as the standard deviation of the stored inter-joint lengths.

[0182] In embodiments of this disclosure, selecting an image combination in operation S230 may include: when multiple image combinations with the same calculated error distance or multiple image combinations with calculated error distances equal to or less than a threshold are identified, in operation S1020, at least two cameras are selected from multiple cameras 111, 112, 113, and 114 based on a preset priority. Selecting an image combination in operation S230 may also include: in S1030, selecting an image combination consisting of at least two images captured and obtained by at least two selected cameras.

[0183] In embodiments of this disclosure, the operation method of the AR device 100 may further include: after obtaining 3D position information of the hand joint from a combination of images of the first image frames, obtaining at least two second image frames by using at least two of a plurality of cameras 111, 112, 113, and 114 configured to obtain at least two images included in the selected image combination. The operation method of the AR device 100 may further include: obtaining 3D position information of the hand joint based on a combination of 2D joint coordinate values ​​obtained from a combination of at least two second image frames.

[0184] In embodiments of this disclosure, selecting an image combination in operation S230 may further include: in operation S1310, identifying image combinations whose calculated error distance exceeds a preset threshold; and in operation S1320, skipping and not performing the calculation of error distances for sub-combinations of the identified image combinations. Selecting an image combination in operation S230 may further include: in operation S1330, selecting the image combination with the smallest error distance from among the image combinations whose error distances have been calculated.

[0185] In embodiments of this disclosure, selecting an image combination in operation S230 may include: in operation S1610, setting a maximum number of images allowed to be combined among a plurality of images; and in operation S1620, calculating the error distance only for image combinations consisting of images whose number is equal to or less than the set maximum number. Selecting an image combination in operation S230 may include: in operation S1630, selecting the image combination with the smallest error distance from the image combinations whose error distances have been calculated.

[0186] In embodiments of this disclosure, obtaining 3D position information of a hand joint in operation S240 may include: additionally selecting at least one 2D joint coordinate value from among 2D joint coordinate values ​​obtained from images other than those included in the selected image combination, that forms an inter-joint length similar to a pre-stored inter-joint length. Obtaining 3D position information of a hand joint in operation S240 may further include: obtaining the 3D position information of the hand joint based on both a combination of 2D joint coordinate values ​​obtained from at least two images included in the selected image combination and the additionally selected at least one 2D joint coordinate value.

[0187] In embodiments of this disclosure, when additionally selecting at least one 2D joint coordinate value, the AR device 100 may select from at least one 2D joint coordinate value obtained from other images a 2D joint coordinate value representing a feature point of a hand joint within the range of joint motion according to the anatomical constraints of the human musculoskeletal system.

[0188] Another aspect of this disclosure provides an AR device 100 for obtaining 3D positional information of a hand joint. According to embodiments of this disclosure, the AR device may include a plurality of cameras 111, 112, 113, and 114 for acquiring multiple images by photographing a user's hand, a memory 130 storing at least one instruction, and at least one processor 120 configured to execute at least one instruction. The at least one processor 120 may execute at least one instruction to obtain 2D joint coordinate values ​​for feature points of the hand joint from the plurality of images acquired by the plurality of cameras 111, 112, 113, and 114. The at least one processor 120 may execute at least one instruction to obtain 3D joint coordinate values ​​of the hand joint based on a combination of 2D joint coordinate values ​​obtained from at least two of the plurality of images. The at least one processor 120 may execute at least one instruction to select an image combination with the smallest error distance from the image combination, the error distance being calculated based on the obtained 3D joint coordinate values. The at least one processor 120 may execute at least one instruction to obtain 3D positional information of the hand joint based on a combination of 2D joint coordinate values ​​from at least two images constituting the selected image combination.

[0189] In embodiments of this disclosure, at least one processor 120 may execute at least one instruction to calculate, when the inter-joint length is not stored in memory 130, the error distance resulting from converting 2D joint coordinate values ​​obtained from at least two images constituting each image combination to 3D joint coordinate values. At least one processor 120 may select the image combination with the smallest calculated error distance from the image combinations.

[0190] In embodiments of this disclosure, at least one processor 120 may execute at least one instruction to obtain information about the center position of a virtual 3D structure formed by rays extending from the center positions of at least two of a plurality of cameras 111, 112, 113, and 114 to 2D joint coordinate values ​​in at least two images. At least one processor 120 may execute at least one instruction to calculate an error distance based on the shortest distance from the center position of the virtual 3D structure to each ray.

[0191] In embodiments of this disclosure, at least one processor 120 may execute at least one instruction to: measure the inter-joint length of the hand based on the obtained 3D joint coordinate values, and calculate an error distance based on the measured inter-joint length and inter-joint length information stored in memory 130. At least one processor 120 may select from a combination of images the one with the smallest calculated error distance.

[0192] In embodiments of this disclosure, at least one processor 120 may execute at least one instruction to select at least two cameras from a plurality of cameras 111, 112, 113, and 114 based on a preset priority when identifying a plurality of image combinations having the same calculated error distance, or when identifying a plurality of image combinations each having a calculated error distance equal to or less than a threshold. At least one processor 120 may execute at least one instruction to select an image combination consisting of at least two images captured and obtained by at least two selected cameras.

[0193] In embodiments of this disclosure, at least one processor 120 may execute at least one instruction to: after obtaining 3D position information of the hand joint from an image combination of first image frames, acquire at least two second image frames by using at least two of a plurality of cameras 111, 112, 113, and 114 configured to acquire at least two images included in the selected image combination. At least one processor 120 may execute at least one instruction to obtain 3D position information of the hand joint based on a combination of 2D joint coordinate values ​​obtained from a combination of at least two second image frames.

[0194] In embodiments of this disclosure, at least one processor 120 may execute at least one instruction to identify image combinations whose calculated error distance exceeds a preset threshold, and skip the calculation of error distances for sub-combinations of the identified image combinations. At least one processor 120 may execute at least one instruction to select the image combination with the smallest error distance from among the image combinations whose error distances have been calculated.

[0195] In embodiments of this disclosure, at least one processor 120 may execute at least one instruction to: set a maximum number of images allowed to be combined among a plurality of images, and calculate error distance only for image combinations consisting of images whose number is equal to or less than the set maximum number. At least one processor 120 may execute at least one instruction to select an image combination with the minimum error distance from among image combinations whose error distances have been calculated.

[0196] Another aspect of this disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions readable by an AR device 100 to cause the AR device 100 to perform the following operations: obtaining 2D joint coordinate values ​​for feature points of the hand joints from multiple images obtained by multiple cameras 111, 112, 113, and 114 capturing a user's hand; obtaining 3D joint coordinate values ​​of the hand joints based on a combination of 2D joint coordinate values ​​obtained from at least two of the multiple images; selecting the image combination with the smallest error distance from the image combination, the error distance being calculated based on the obtained 3D joint coordinate values; and obtaining 3D position information of the hand joints based on a combination of 2D joint coordinate values ​​from at least two images constituting the selected image combination.

[0197] The program executed by the AR device 100 as described in this disclosure can be implemented using hardware elements, software elements, and / or combinations thereof. The program can be executed by any system capable of executing computer-readable instructions.

[0198] Software may include computer programs, code, instructions, or one or more combinations thereof, and may configure a processing device to operate as desired, or to instruct a processor to operate independently or collectively.

[0199] Software can be implemented using a computer program that includes instructions stored in a computer-readable recording (or storage) medium. Examples of computer-readable recording media include magnetic storage media (e.g., read-only memory (ROM), floppy disks, hard disks, etc.) and optical recording media (e.g., compact disc ROM (CD-ROM) or digital versatile disc (DVD)). Computer-readable recording media can also be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. The medium can be read by a computer, stored in memory, and executed by a processor.

[0200] Computer-readable storage media may be provided in the form of non-transitory storage media. The term "non-transitory" means that the storage medium is tangible and does not include signals, but does not distinguish between any data that is semi-permanently or temporarily stored in the storage medium. For example, a non-transitory storage medium may include a buffer that temporarily stores data.

[0201] Furthermore, programs according to embodiments of this disclosure may be provided in a computer program product. The computer program product may be a commercial product that can be traded between a seller and a buyer.

[0202] Computer program products may include software programs and computer-readable storage media on which the software programs are stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by the manufacturer of AR device 100 or an electronic marketplace (e.g., Samsung Galaxy Store®). For electronic distribution, at least a portion of the software program may be stored in a storage medium or temporarily generated. In this case, the storage medium may be one of the AR device 100 manufacturer's server, an electronic marketplace server, or a relay server temporarily storing the software program.

[0203] In a system including AR device 100 and / or a server, the computer program product may include the storage medium of the server or the storage medium of AR device 100. Alternatively, when a third device (e.g., a mobile device) communicatively connected to AR device 100 is present, the computer program product may include the storage medium of the third device. In another example, the computer program product may include the software program itself transmitted from AR device 100 to a third device or from a third device to an electronic device.

[0204] In this scenario, one of the AR device 100 and the third device may execute a computer program product to perform the method according to embodiments of the present disclosure. Alternatively, at least one of the AR device 100 and the third device may execute a computer program product to perform the method according to embodiments of the present disclosure in a distributed manner.

[0205] For example, AR device 100 can execute functions stored in memory 130 (see...). Figure 3 The computer program product in the present disclosure is used to control another electronic device communicatively connected to the AR device 100 to perform the method according to the embodiments of the present disclosure.

[0206] In another example, a third device may execute a computer program product to control an electronic device communicatively connected to the third device to perform a method according to an embodiment of the present disclosure.

[0207] When a third device executes a computer program product, the third device can download the computer program product from AR device 100 and execute the downloaded computer program product. Alternatively, the third device can execute a pre-loaded computer program product to perform a method according to an embodiment of this disclosure.

[0208] Although this disclosure has been described with reference to some embodiments and accompanying drawings, it will be apparent to those skilled in the art that various modifications and changes can be made to the embodiments. For example, the foregoing methods may be performed in a different order, and / or the foregoing components, such as computer systems or modules, may be combined in a form different from that described above, and / or replaced or substituted by other components or their equivalents to obtain appropriate results.

Claims

1. A method for an augmented reality (AR) device (100) to obtain three-dimensional 3D position information of a hand joint, the method comprising: Two-dimensional 2D joint coordinate values ​​for feature points of the hand joints are obtained from multiple images of the user's hand captured by multiple cameras (111, 112, 113, 114) (S210). The 3D joint coordinate values ​​of the hand joint are estimated based on the combination of the 2D joint coordinate values ​​obtained from the combination of at least two images from the plurality of images (S220). Select the image combination with the smallest error distance from the image combinations (S230), where the error distance is calculated based on the estimated 3D joint coordinate values; as well as Based on a combination of 2D joint coordinate values ​​from at least two images constituting the selected image combination, 3D position information of the hand joint is obtained (S240).

2. The method according to claim 1, wherein, Selecting the image combination (S230) includes: When the length between the hand joints is not stored in memory, the error distance generated when converting the 2D joint coordinate values ​​obtained from at least two images constituting each of the image combination to the 3D joint coordinate values ​​is calculated (S520); and Select the image combination with the smallest calculated error distance from the image combinations (S530).

3. The method according to claim 1, wherein, Selecting the image combination (S230) includes: Based on the estimated 3D joint coordinate values, the length between the hand joints is measured (S810). Based on the measured lengths between the hand joints and the information about the lengths between the joints stored in the memory, the error distance is calculated (S820); and Select the image combination with the smallest calculated error distance from the image combinations (S560).

4. The method according to any one of claims 1 to 3, wherein, Selecting the image combination (S230) includes: When multiple image combinations with the same or a calculated error distance equal to or less than a threshold are identified, at least two cameras are selected from the plurality of cameras (111, 112, 113, 114) based on a preset priority (S1020); and Select an image combination consisting of at least two images captured and obtained by at least two selected cameras (S1030).

5. The method according to any one of claims 1 to 4, further comprising: After obtaining the 3D position information of the hand joint from the image combination of the first image frame, at least two second image frames are obtained by using at least two of the plurality of cameras (111, 112, 113, 114) configured to obtain at least two images included in the selected image combination. as well as The 3D position information of the hand joint is obtained based on the combination of 2D joint coordinate values ​​obtained from the combination of the at least two second image frames.

6. The method according to any one of claims 1 to 5, wherein, Selecting the image combination (S230) includes: Identify image combinations whose calculated error distance exceeds a preset threshold (S1310). Skip and do not perform the calculation of the error distance for the sub-combinations of the identified image combinations (S1320); and Select the image combination with the smallest calculated error distance from the image combinations that have already calculated the error distance (S1330).

7. The method according to any one of claims 1 to 6, wherein, Selecting the image combination (S230) includes: Set the maximum number of images that can be combined among the plurality of images (S1610). The error distance is calculated only for image combinations consisting of images whose number is equal to or less than the set maximum number (S1620); and Select the image combination with the smallest error distance from the image combinations whose error distance has been calculated (S1630).

8. An augmented reality (AR) device (100) for obtaining three-dimensional (3D) positional information of a hand joint, the AR device (100) comprising: Multiple cameras (111, 112, 113, 114) are configured to capture multiple images by photographing the user's hand; Memory (130) stores at least one instruction; as well as At least one processor (120) is configured to execute the at least one instruction. The at least one processor (120) is configured to: Two-dimensional 2D joint coordinate values ​​for feature points of the hand joint are obtained from multiple images acquired through the multiple cameras (111, 112, 113, 114). The 3D joint coordinates of the hand joint are estimated based on a combination of 2D joint coordinate values ​​obtained from image combinations consisting of at least two images from the plurality of images. The image combination with the smallest error distance is selected from the image combinations, the error distance being calculated based on the estimated 3D joint coordinate values. The 3D position information of the hand joint is obtained based on a combination of 2D joint coordinate values ​​from at least two images constituting the selected image combination.

9. The AR device (100) according to claim 8, wherein, The at least one processor (120) is configured to execute the at least one instruction to: When the length between the hand joints is not stored in the memory (130), the error distance generated when converting the 2D joint coordinate values ​​obtained from at least two images constituting each of the image combination to the 3D joint coordinate values ​​is calculated, and Select the image combination with the smallest calculated error distance from the image combinations.

10. The AR device (100) according to claim 8, wherein, The at least one processor (120) is configured to execute the at least one instruction to: Based on the estimated 3D joint coordinate values, the length between the hand joints is measured. Based on the measured lengths between the hand joints and the information about the lengths between the joints stored in the memory (130), the error distance is calculated, and Select the image combination with the smallest calculated error distance from the image combinations.

11. The AR device (100) according to any one of claims 8 to 10, wherein, The at least one processor (120) is configured to execute the at least one instruction to: When multiple image combinations with the same or a calculated error distance equal to or less than a threshold are identified, at least two cameras are selected from the plurality of cameras (111, 112, 113, 114) based on a preset priority. Select an image combination consisting of at least two images captured and obtained by at least two selected cameras.

12. The AR device (100) according to any one of claims 8 to 11, wherein, The at least one processor (120) is configured to execute the at least one instruction to: After obtaining the 3D position information of the hand joint from the image combination of the first image frames, at least two second image frames are obtained by using at least two of the plurality of cameras (111, 112, 113, 114) configured to obtain at least two images included in the selected image combination, and The 3D position information of the hand joint is obtained based on the combination of 2D joint coordinate values ​​obtained from the combination of the at least two second image frames.

13. The AR device (100) according to any one of claims 8 to 12, wherein, The at least one processor (120) is configured to execute the at least one instruction to: The system identifies image combinations whose calculated error distance exceeds a preset threshold, and skips and does not perform error distance calculations for sub-combinations of the identified image combinations. Select the image combination with the smallest calculated error distance from the image combinations whose error distance has been calculated.

14. The AR device (100) according to any one of claims 8 to 13, wherein, The at least one processor (120) is configured to execute the at least one instruction to: Set the maximum number of images that can be combined among the multiple images. The error distance is calculated only for image combinations consisting of a number equal to or less than the set maximum number of images. Select the image combination with the smallest calculated error distance from the image combinations whose error distance has been calculated.

15. A computer program product comprising a computer-readable storage medium, wherein, The computer-readable storage medium includes instructions for a method of obtaining three-dimensional 3D position information of a hand joint using an augmented reality (AR) device (100), the method comprising: Two-dimensional 2D joint coordinate values ​​for feature points of the hand joints are obtained from multiple images of the user's hand captured by multiple cameras (111, 112, 113, 114). The three-dimensional 3D joint coordinate values ​​of the hand joint are estimated based on the combination of the 2D joint coordinate values ​​obtained from the combination of at least two images from the plurality of images. The image combination with the smallest error distance is selected from the image combinations, the error distance being calculated based on the estimated 3D joint coordinate values; and The 3D position information of the hand joint is obtained based on a combination of 2D joint coordinate values ​​from at least two images constituting the selected image combination.

Images