Method and apparatus for recogniting text based on hand interaction for ar glasses

Abstract

A hand interaction-based text recognition method for AR glasses according to one embodiment of the present invention may include the steps of collecting RGB images, extracting hand joint information from the RGB images, generating a text image based on the hand joint information, recognizing text in the text image, and outputting the recognized text.

Description

Technology Field

[0001] The present invention relates to a method and apparatus for recognizing text through hand interaction on Augmented Reality (AR) content. Background Technology

[0002] Currently commercialized augmented reality-based text input systems mainly consist of methods where text is entered by manipulating a virtual input keyboard created on a UI, such as Microsoft’s HoloLens, or methods where text is entered using an actual device (such as a mobile phone), such as Nreal’s Nreal Light. When using a virtual input device like HoloLens, there are problems such as poor quality of hand interaction or frequent input errors if the size of the UI (user interface) is insufficient, and when using an actual device like Nreal Light, there is a problem that the device must be carried at all times.

[0003] Among them, when using a virtual input device, the quality of hand interaction in a head-mounted display (HMD) type device that includes its own computing device and multiple sensors does not decrease relatively, but in a glass type device, the quality of hand interaction decreases, leading to frequent input errors. The problem to be solved

[0004] The objective of the present invention is to provide a text recognition method and device for solving the problems of mis-input and mis-recognition caused by low hand interaction quality in an Android-based glass device with limited computing resources. means of solving the problem

[0005] A hand interaction-based text recognition method for AR glasses according to an embodiment for achieving the above-mentioned purpose may include the steps of collecting RGB images, extracting hand joint information from the RGB images, generating a text image based on the hand joint information, recognizing text in the text image, and outputting the recognized text.

[0006] The step of extracting hand joint information may include a step of estimating hand motion from the RGB image and a step of extracting hand joint information based on the hand motion.

[0007] The step of generating the text image may include the step of extracting raycast point values ​​based on the hand joint information, the step of generating an output value by performing hand tremor correction based on the raycast point values ​​and rendering a line on a canvas UI based on the output value, and the step of generating texture image information by converting the rendered canvas UI into a rendering image texture.

[0008] The above hand tremor correction comprises the steps of inputting the raycast point values ​​into a raycast point queue, removing the first k points and the last k points among the raycast point values ​​input into the raycast point queue, and

[0009] It may include a step of outputting an output value by applying the moving average of the raycast point queue from which the first k points and the last k points have been removed.

[0010] The step of recognizing the text may include a step of preprocessing the text image and a step of inferring text using the preprocessed text image as input to a first machine learning model.

[0011] The step of preprocessing the text image may include converting the text image into a grayscale image, adjusting the resolution of the grayscale image, normalizing the pixel values ​​of the resolution-adjusted image to have values ​​from 0 to 1, and converting the normalized image into a tensor.

[0012] The step of inferring the text may include the step of extracting structural features of the text by using the preprocessed text image as input to a second machine learning model, the step of converting the structural features of the text into a one-dimensional text dimension, the step of extracting segment-specific features by using the features converted to the one dimension as input to a third machine learning model, and the step of inferring the text based on the segment-specific features.

[0013] The above second machine learning model may be composed of two-dimensional convolutional layers, and the above third machine learning model may be composed of one-dimensional convolutional layers.

[0014] The step of converting the structural features of the above text into a one-dimensional text dimension can be performed using the average, maximum, and variance values ​​of the above features.

[0015] The step of inferring text based on the above-mentioned segment-specific features can be performed based on the Dynamic Time Warping (DTW) loss function and the Connectionist Temporal Classification (CTC) loss function.

[0016] In addition, a hand interaction-based text recognition device for AR glasses according to an embodiment for achieving the above-mentioned purpose includes a memory in which a control program for text recognition is stored, and a processor that executes the control program stored in the memory. The processor can collect an RGB image, extract hand joint information from the RGB image, generate a text image based on the hand joint information, recognize text in the text image, and output the recognized text.

[0017] The above processor can extract hand joint information, estimate hand movements from the RGB image, and extract hand joint information based on the hand movements.

[0018] The processor can extract raycast point values ​​based on the hand joint information, perform hand tremor correction based on the raycast point values ​​to generate output values, render lines on a canvas UI based on the output values, and convert the rendered canvas UI into a rendering image texture to generate texture image information.

[0019] The processor can correct hand tremor by inputting the raycast point values ​​into a raycast point queue, removing the first k points and the last k points among the raycast point values ​​input into the raycast point queue, and applying the moving average of the raycast point queue from which the first k points and the last k points have been removed to output an output value.

[0020] The processor can preprocess the text image and infer text using the preprocessed text image as input to a first machine learning model.

[0021] The processor can convert a text image into a grayscale image, adjust the resolution of the grayscale image, normalize the pixel values ​​of the resolution-adjusted image to have values ​​from 0 to 1, and convert the normalized image into a tensor.

[0022] The processor can extract structural features of text by using the preprocessed text image as input to a second machine learning model, convert the structural features of text into a one-dimensional text dimension, extract segment-specific features by using the features converted into the one dimension as input to a third machine learning model, and infer text based on the segment-specific features.

[0023] The above second machine learning model may be composed of two-dimensional convolutional layers, and the above third machine learning model may be composed of one-dimensional convolutional layers.

[0024] The processor can convert the structural features of the text into a one-dimensional text dimension using the average, maximum, and variance values ​​of the features.

[0025] The above processor can perform an operation of inferring text based on the above interval features based on the DTW (Dynamic Time Warping) loss function and the CTC (Connectionist Temporal Classification) loss function. Effects of the invention

[0026] The embodiment enables stable text input by reducing input errors through the use of a deep learning-based optical character recognition device and a hand tremor correction method.

[0027] In addition, the embodiment can improve processing speed and accuracy by using a machine learning model of a convolutional layer.

[0028] In addition, the embodiment can be applied to mobile and AR glass environments. Brief explanation of the drawing

[0029] FIG. 1 is a block diagram showing a hand interaction-based text recognition device for AR glasses according to an embodiment. FIG. 2 is a block diagram showing the hand interaction part of a hand interaction-based text recognition device for AR glasses according to an embodiment. FIG. 3 is a block diagram showing a text generation unit of a hand interaction-based text recognition device for AR glasses according to an embodiment. FIG. 4 is a diagram showing the operation performed in the text generation unit of a hand interaction-based text recognition device for AR glasses according to an embodiment. Figure 5 is a graph illustrating the noise removal process of a signal using a moving average according to an embodiment. FIG. 6 is a diagram showing the input results according to the presence or absence of hand tremor correction according to an embodiment. FIG. 7 is a block diagram showing a text recognition unit of a hand interaction-based text recognition device for AR glasses according to an embodiment. FIG. 8 is a diagram showing a machine learning model used in a text recognition unit according to an embodiment. FIG. 9 is a diagram showing the feature synthesis process performed in the text recognition unit according to the embodiment. FIG. 10 is a diagram showing the segment feature extraction and text classification process performed in the text recognition unit according to the embodiment. FIG. 11 is a diagram showing the result recognized by the text recognition unit according to the embodiment. FIG. 12 is a flowchart illustrating a hand interaction-based text recognition method for AR glasses according to an embodiment. FIG. 13 is a block diagram showing the configuration of a computer system according to an embodiment. Specific details for implementing the invention

[0030] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0031] Although terms such as "first" or "second" are used to describe various components, these components are not limited by such terms. Such terms may be used merely to distinguish one component from another. Accordingly, the first component mentioned below may be the second component within the technical scope of the present invention.

[0032] The terms used herein are for describing embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used in this specification, "comprises" or "comprising" implies that the mentioned component or step does not exclude the presence or addition of one or more other components or steps.

[0033] Unless otherwise defined, all terms used in this specification may be interpreted in a sense that is commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0034] In this document, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “at least one of A, B or C,” and “at least one of A, B, or C” may include any one of the items listed together with the corresponding phrase, or all possible combinations thereof.

[0035] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0036] FIG. 1 is a block diagram showing a hand interaction-based text recognition device for AR glasses according to an embodiment.

[0037] Referring to FIG. 1, a text recognition device according to an embodiment may include an input unit (100), a hand interaction unit (200), a text generation unit (300), a text recognition unit (400), and an output unit (500).

[0038] The input unit (100) can receive RGB images from the front camera of the Nreal Glass. The hand interaction unit (200) can estimate hand movements from the RGB image input and extract hand joint information. The text generation unit (300) can generate a text image to be used for recognition in the text generation unit using the extracted joint trajectories. The text recognition unit (400) can recognize text based on the generated text image. The output unit can output the recognized text to the Nreal Glass.

[0039] FIG. 2 is a block diagram showing the hand interaction part of a hand interaction-based text recognition device for AR glasses according to an embodiment.

[0040] As illustrated in FIG. 2, the hand interaction unit (200) according to the embodiment can preprocess an input RGB image. The hand interaction unit (200) according to the embodiment can estimate hand movements based on the preprocessed image. Here, a machine learning model can be used for the hand movements. The machine learning model may be a RetinaFace model, but its type is not limited. The hand interaction unit (200) according to the embodiment can extract hand joint information based on the hand movements.

[0041] FIG. 3 is a block diagram showing a text generation unit of a hand interaction-based text recognition device for AR glasses according to an embodiment.

[0042] As illustrated in FIG. 3, the text generation unit (300) according to the embodiment may include an interface (310) and a hand tremor corrector (320).

[0043] The interface (310) can input text by interacting with the UI through the extracted hand joints. The hand tremor corrector (320) can compensate for the problem of incorrect input occurring during text input.

[0044] The interface (310) can use a raycast pointer for text input. The raycast pointer can be projected from the arm toward the wrist.

[0045] FIG. 4 is a diagram showing the operation performed in the text generation unit of a hand interaction-based text recognition device for AR glasses according to an embodiment.

[0046] As shown in FIG. 4, input can be started by performing the Grab gesture while the Grab gesture is released, and input can be ended by releasing the Grab gesture.

[0047] Returning to FIG. 3, the interface (310) can obtain the coordinates of the part where the raycast pointer touches when input begins on the canvas UI. The hand tremor corrector (320) can put these coordinate values ​​into a raycast point queue and then perform hand tremor correction. The interface (310) can use the hand tremor correction results to draw a line on the canvas UI and perform the process of rendering a line.

[0048] The interface (310) can convert the canvas UI with the drawn lines into a RenderTexture using the Unity Camera to finally obtain texture image information.

[0049] When text input is performed in the text generator (310), slight shaking occurs due to the frame-by-frame hand pose estimation error, and due to this shaking, the quality of the text converted into an image is poor, which may cause misrecognition in the text recognition device. Therefore, if the quality of the hand interaction is poor due to the device's performance or algorithm, or if it is far from the input UI, this problem occurs easily.

[0050] In addition, errors in input occur, such as the addition of unnecessary strokes, during the process of changing gestures to control the start and end of input during interaction. To prevent such errors in input, hand tremor correction is applied to the text input area.

[0051] First, to mitigate input errors caused by gesture changes during interaction, the first and last K points corresponding to the change process are removed. Second, a moving average can be applied to prevent input errors caused by hand pose estimation errors.

[0052] Figure 5 is a graph illustrating the noise removal process of a signal using a moving average according to an embodiment.

[0053] As shown in Figure 5, applying a moving average changes the value to the average of nearby signals, thereby removing noise.

[0054] FIG. 6 is a diagram showing the input results according to the presence or absence of hand tremor correction according to an embodiment.

[0055] As shown in Fig. 6, when hand shake correction is not applied, the quality of the text image deteriorates, causing problems in text recognition. On the other hand, when hand shake correction is applied, the quality of the text image improves, allowing for effective text recognition.

[0056] FIG. 7 is a block diagram showing a text recognition unit of a hand interaction-based text recognition device for AR glasses according to an embodiment.

[0057] Referring to FIG. 7, the text recognition unit (400) according to the embodiment may include an image processor (410) and a text inference unit (420).

[0058] The image processing unit (410) can preprocess the input image so that it can be used as input to a machine learning model. The text inferor (420) can recognize text using the machine learning model learned from the preprocessed image.

[0059] The machine learning model of the text inferer (420) can only accept images of a specified format. A preprocessing process is required to pass the input images to the text inferer (420), and this preprocessing process can be performed by an image processor.

[0060] The image processor (410) can perform four operational processes. First, the input 3-channel RGB image can be converted into a single-channel grayscale image, and then the resolution of the image can be adjusted to a size that can be input into a machine learning model. Afterward, the image can be normalized so that the pixel values ​​have values ​​between 0 and 1. The normalized image can finally be converted into a tensor used for inference. The converted tensor can be used as input to a text inferencing machine.

[0061] The text inferencing unit (420) can be composed of machine learning models and can be divided into four types according to their roles. The text inferencing unit (420) can extract image features, synthesize (or combine) features, extract segment features, and classify text.

[0062] FIG. 8 is a diagram showing a machine learning model used in a text recognition unit according to an embodiment.

[0063] As illustrated in Fig. 8, the machine learning model for image feature extraction consists of 2D convolutional layers and can perform the role of extracting structural features of text. The machine learning model utilizes a 3-Stage structure using a DenseNet architecture, capable of extracting features by synthesizing information ranging from simple data such as color to straight lines, curves, and structural information. An attention module is inserted between each layer to remove unimportant information such as noise. Additionally, assuming smooth operation in an AR environment, the convolutional layers are separated into pointwise convolutional layers and depthwise convolutional layers to reduce the total size of the model.

[0064] Returning to Fig. 7, the text inferor (420) can synthesize features.

[0065] FIG. 9 is a diagram showing the feature synthesis process performed in the text recognition unit according to the embodiment.

[0066] As illustrated in FIG. 9, the text inferer (420) of the text recognition unit can synthesize two-dimensional features and convert them into one-dimensional text. At this time, three statistical values—average, maximum, and variance—are used to synthesize the features. Conventionally, the average or a combination of the average and maximum is used when synthesizing, but variance is added to increase the amount of information preserved. When configured in this way, the maximum, average, and variance values ​​are synthesized for each pixel, and these three values ​​can be used to finally synthesize them into a single feature. At this time, a point-by-point convolution layer can be used to synthesize the features.

[0067] Returning to Fig. 7, the text inferor (420) can extract interval features. The interval features can utilize a machine learning model composed of one-dimensional convolutional layers, and the interval features are extracted by combining nearby structural features.

[0068] Interval feature extraction is divided into short-term, mid-term, and long-term modeling depending on the length of the processing area; while longer processing areas are suitable for high-resolution image processing, the inference speed decreases somewhat.

[0069] Interval feature extraction can determine the intervals by pre-inputting the kernel size of the machine learning model.

[0070] The text inferer (420) can classify text. Text classification can be performed using a machine learning model. The machine learning model consists of one-dimensional convolutional layers and can ultimately generate text through segment classification from extracted features.

[0071] FIG. 10 is a diagram showing the segment feature extraction and text classification process performed in the text recognition unit according to the embodiment.

[0072] As illustrated in Fig. 10, segment feature extraction and text classification can be configured into three types to achieve consistent output at various input resolutions. Segment features are divided into short-term modeling, mid-term modeling, and long-term modeling. Short-term modeling uses only a narrow region to classify text, so the inference speed is fast, but accuracy decreases as the resolution increases. In the case of long-term modeling, a wide region is used to classify text, so the inference speed is slower, but it demonstrates good performance even at high resolutions.

[0073] Since all three models must produce the same output for the same input, the loss function can be configured to produce identical outputs. However, because each model has a different output length, the Dynamic Time Warping (DTW) loss function used in the field of signal processing can be utilized. This implementation uses soft-DTW.

[0074] DTW expresses signal similarity in the form of distance. Unlike the conventional Euclidean distance, it allows for comparison even if the lengths of the two objects being compared differ and the timing of the comparison does not necessarily coincide. Comparisons are performed using three combinations: DTW(Short, Mid), DTW(Short, Long), and DTW(Mid, Long). The sum of these three combinations is used as the loss. When the loss function is constructed in this way, training proceeds as shown in Equation 1 so that each output produces a similar output.

[0075] [Mathematical Formula 1]

[0076]

[0077] Here, is a set of possible paths, and is a function representing the similarity of paths. Here, a path refers to a combination of feature points of A and feature points of B, assuming that features A and B are compared.

[0078] To classify text, Connectionist Temporal Classification (CTC) is used as the loss function that compares the Ground Truth (GT) with the output result. This CTC applies Equation 2 individually to the three parts: Short-term modeling, Mid-term modeling, and Long-term modeling.

[0079] [Mathematical Formula 2]

[0080]

[0081] Here, is a set of possible paths, and represents the conditional probability of the t-th feature point.

[0082] In this case, training using DTW loss has the characteristic of normalizing the output of the segment feature extraction unit, which hinders learning in the early stages; therefore, it is applied after sufficient training has been performed using CTC loss. This is because the goal of this model is to ensure that the DTW loss produces the same output, so applying it before a certain level of training has been completed results in low performance on the actual target text classification problem. Therefore, to address this issue, a separate scheduler is used for the DTW loss. As shown in Equation 3, for DTW It is multiplied by, and this value starts at 0 at the start of learning and gradually increases as learning progresses.

[0083] [Mathematical Formula 3]

[0084]

[0085] Applying DTW loss to the model can resolve the discrepancy in output between long-term and short-term models that occurs when using only CTC loss, and enables stable output from inputs of various resolutions. The convolutional layer can recognize only fixed-size regions, and is composed of three models based on region size; training is then performed so that the three models produce the same output from the same input.

[0086] The finally trained model selects segment feature extraction based on the objective. If the input image resolution is low, a model applying short-term modeling is used; if it is high, long-term modeling; and if it is variable resolution, mid-term modeling. If long-term modeling is applied to low resolution, the recognition range per character is wide, which can lead to cases where multiple characters are recognized as a single character; therefore, selecting an appropriate model is necessary to achieve optimal accuracy.

[0087] FIG. 11 is a diagram showing the result recognized by the text recognition unit according to the embodiment.

[0088] As shown in FIG. 11, compared to the conventional text recognition result (left), it can be seen that the text recognition result according to the embodiment (right) is recognized more effectively.

[0089] FIG. 12 is a flowchart illustrating a hand interaction-based text recognition method for AR glasses according to an embodiment.

[0090] As illustrated in FIG. 12, a hand interaction-based text recognition method for AR glasses according to an embodiment may include a step of collecting RGB images (S100), a step of extracting hand joint information (S200), a step of generating a text image (S300), a step of recognizing text (S400), and a step of outputting text (S500). Here, the hand interaction-based text recognition method for AR glasses according to an embodiment may be performed in a text recognition device.

[0091] A text recognition device according to an embodiment can collect RGB images from the front camera of the Nreal Glass (S100). A text recognition device according to an embodiment can estimate hand movements from the RGB images. A text recognition device according to an embodiment can extract hand joint information based on hand movements (S200).

[0092] The text recognition device according to the embodiment can generate a text image based on hand joint information (S300). The text recognition device according to the embodiment can perform hand tremor correction during the process of generating the text image.

[0093] A text recognition device according to an embodiment can recognize text in a generated text image (S400). The text recognition device according to an embodiment can preprocess a text image and infer text from the preprocessed text image using a machine learning model.

[0094] The text recognition device according to the embodiment can output the recognized text to Nreal Glass (S500).

[0095] A hand interaction-based text recognition device for AR glasses according to an embodiment can be implemented in a computer system such as a computer-readable recording medium.

[0096] FIG. 13 is a block diagram showing the configuration of a computer system according to an embodiment.

[0097] Referring to FIG. 13, a computer system (1000) according to an embodiment may include one or more processors (1010), memory (1030), user interface input device (1040), user interface output device (1050), and storage (1060) that communicate with each other through a bus (1020). Additionally, the computer system (1000) may further include a network interface (1070) connected to a network.

[0098] The processor (1010) may be a semiconductor device that executes programs or processing instructions stored in memory or storage, or a central processing unit. The processor (1010) may be a type of central processing unit and may control the entire operation of a hand interaction-based text recognition device for AR glasses.

[0099] The processor (1010) may include all types of devices capable of processing data. Here, 'processor' may refer to a data processing device embedded in hardware that has a physically structured circuit to perform functions expressed by code or instructions included in a program, for example. Examples of such data processing devices embedded in hardware may include, but are not limited to, processing devices such as a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA).

[0100] The memory (1030) may store various data for overall operation, such as a control program for performing a hand interaction-based text recognition method for AR glasses according to the embodiment. Specifically, the memory may store a number of applications running on a hand interaction-based text recognition device for AR glasses, and data and commands for the operation of the hand interaction-based text recognition device for AR glasses.

[0101] The memory (1030) and storage (1060) may be storage media comprising at least one of a volatile medium, a non-volatile medium, a removable medium, a non-removable medium, a communication medium, or an information transfer medium. For example, the memory (1030) may include a ROM (1031) or a RAM (1032).

[0102] The specific embodiments described in this invention are examples and do not limit the scope of the invention in any way. For the sake of brevity of the specification, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted. Additionally, the connections of lines or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be replaced or additionally represented as various functional connections, physical connections, or circuit connections in actual devices. Furthermore, unless specifically stated with terms such as "essential" or "importantly," a component may not be strictly necessary for the application of the invention.

[0103] Accordingly, the scope of the present invention should not be limited to the embodiments described above, and all scopes equivalent to or equivalently modified from the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the concept of the present invention. Explanation of the symbols

[0104] 100: Input section 200: Hand Interaction Section 300: Text generation section 400: Text recognition unit 500: Output section

Claims

Claim 1 A text recognition method comprising: a step of collecting an RGB image; a step of extracting hand joint information from the RGB image; a step of generating a text image based on the hand joint information; a step of recognizing text in the text image; and a step of outputting the recognized text; wherein the step of recognizing text includes a step of preprocessing the text image; and a step of inferring text using the preprocessed text image as input to a first machine learning model; wherein the step of inferring text includes a step of extracting structural features of text using the preprocessed text image as input to a second machine learning model; a step of converting the structural features of text into a one-dimensional text dimension; a step of extracting segment-specific features using the features converted to the one dimension as input to a third machine learning model; and a step of inferring text based on the segment-specific features. Claim 2 A text recognition method according to claim 1, wherein the step of extracting hand joint information comprises: a step of estimating hand motion from the RGB image; and a step of extracting hand joint information based on the hand motion. Claim 3 A text recognition method according to claim 1, wherein the step of generating the text image comprises: a step of extracting raycast point values ​​based on the hand joint information; a step of generating an output value by performing hand tremor correction based on the raycast point values ​​and rendering a line on a canvas UI based on the output value; and a step of converting the rendered canvas UI into a rendering image texture to generate texture image information. Claim 4 In paragraph 3, the hand tremor correction comprises: a step of inputting the raycast point values ​​into a raycast point queue; a step of removing the first k points and the last k points among the raycast point values ​​input into the raycast point queue; and a step of outputting an output value by applying a moving average of the raycast point queue from which the first k points and the last k points have been removed. Claim 5 delete Claim 6 A text recognition method according to claim 1, wherein the step of preprocessing the text image comprises: a step of converting the text image into a grayscale image; a step of adjusting the resolution of the grayscale image; a step of normalizing the pixel values ​​of the resolution-adjusted image to have values ​​from 0 to 1; and a step of converting the normalized image into a tensor. Claim 7 delete Claim 8 A text recognition method according to claim 1, wherein the second machine learning model is composed of two-dimensional convolutional layers and the third machine learning model is composed of one-dimensional convolutional layers. Claim 9 In claim 1, the step of converting the structural features of the text into a one-dimensional text dimension is a text recognition method performed using the average value, maximum value, and variance value of the features. Claim 10 In claim 1, the step of inferring text based on the features of each segment is a text recognition method performed based on a Dynamic Time Warping (DTW) loss function and a Connectionist Temporal Classification (CTC) loss function. Claim 11 A text recognition device comprising: a memory storing a control program for text recognition; and a processor executing the control program stored in the memory, wherein the processor collects an RGB image, extracts hand joint information from the RGB image, generates a text image based on the hand joint information, recognizes text in the text image, and outputs the recognized text, wherein the processor preprocesses the text image and infers text using the preprocessed text image as input to a first machine learning model, wherein the processor extracts structural features of text using the preprocessed text image as input to a second machine learning model, converts the structural features of text into a one-dimensional text dimension, extracts segment-specific features using the features converted to the one dimension as input to a third machine learning model, and infers text based on the segment-specific features. Claim 12 In claim 11, the processor is a text recognition device that extracts hand joint information, estimates hand motion from RGB images, and extracts hand joint information based on hand motion. Claim 13 In claim 11, the processor extracts a raycast point value based on the hand joint information, performs hand tremor correction based on the raycast point value to generate an output value, renders a line on a canvas UI based on the output value, and converts the rendered canvas UI into a rendering image texture to generate texture image information. Claim 14 A text recognition device that corrects hand tremor in claim 13, wherein the processor inputs the raycast point values ​​into a raycast point queue, removes the first k points and the last k points among the raycast point values ​​input into the raycast point queue, and outputs an output value by applying the moving average of the raycast point queue from which the first k points and the last k points have been removed. Claim 15 delete Claim 16 In claim 11, the processor converts a text image into a grayscale image, adjusts the resolution of the grayscale image, normalizes the pixel values ​​of the resolution-adjusted image to have values ​​from 0 to 1, and converts the normalized image into a tensor. Claim 17 delete Claim 18 A text recognition device according to claim 11, wherein the second machine learning model is composed of two-dimensional convolutional layers and the third machine learning model is composed of one-dimensional convolutional layers. Claim 19 In claim 11, the processor is a text recognition device that converts the structural features of the text into a one-dimensional text dimension using the average value, maximum value, and variance value of the features. Claim 20 In claim 11, the processor is a text recognition device that performs the operation of inferring text based on the segment-specific features based on the DTW (Dynamic Time Warping) loss function and the CTC (Connectionist Temporal Classification) loss function.

Images