Method and apparatus for interacting with user in multi-layer structure

Abstract

The present specification relates to a method by which a user terminal interacts with a user in a multi-layer structure, comprising the steps of: generating the multi-layer structure on the basis of image files corresponding to each page of a reference book, wherein the multi-layer structure includes 1) a contents layer, 2) an interaction layer, and 3) a canvas layer; and generating an interaction object for an interaction with the user, in an interaction layer corresponding to the current page, on the basis of area information and question information of the current page displayed to the user.

Description

Method and device for interacting with users in a multi-layer structure

[0001] The present specification relates to a method and a device for interacting with a user in a multi-layer structure for displaying the contents of a reference book.

[0002] Most paper reference books are available in printed form, allowing users to write down questions and answers, erasing them and re-solving them if necessary. This method is relatively simple, but it offers the advantage of allowing users to solve problems anytime, anywhere, as long as they have paper and a pencil.

[0003] Recently, with the advancement of electronic devices, it has become popular to provide reference books in digital form using devices such as tablet PCs and smartphones, and for users to solve problems using devices such as touch screens or styluses.

[0004] Technology that utilizes these electronic devices to provide reference book content can offer users several advantages. For example, unlike printed reference books, reference books provided digitally can be easily accessed anytime, anywhere. Furthermore, providing problems in digital format allows users to immediately check the answers, allowing them to correct errors in real time while solving the problems, enabling them to study more efficiently.

[0005] Additionally, devices like tablet PCs and smartphones allow students to solve problems using devices like touchscreens or styluses. Unlike paper and pencil, this method offers additional functionality and makes the problem-solving process more convenient.

[0006] In particular, these technologies are being widely utilized in the education sector. Schools and academies are providing reference books using electronic devices or operating learning programs that utilize them to enhance student learning efficiency. Furthermore, smartphones and tablet PCs are being utilized in diverse fields, including language learning, foreign language learning, and mathematics learning.

[0007] The purpose of this specification is to propose a method for interacting with users in a multi-layer structure for providing the contents of a reference book.

[0008] Additionally, the purpose of this specification is to propose a method for resolving issues arising from overlap between the Interaction Layer and the Canvas Layer of a multi-layer structure.

[0009] The technical problems to be solved by this specification are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which this specification belongs from the detailed description of the specification below.

[0010] One aspect of the present specification provides a method for a user terminal to interact with a user in a multi-layer structure, the method comprising: generating the multi-layer structure based on image files corresponding to each page of a reference book; wherein the multi-layer structure includes 1) a content layer, 2) an interaction layer, and 3) a canvas layer; and generating an interaction object for interaction with the user in the interaction layer corresponding to the current page based on area information and question information of the current page displayed to the user. The method may further include: detecting an interaction with the user based on the interaction object; and transmitting a detected event to the interaction layer and processing it based on the detected interaction.

[0011] In addition, the step of detecting interaction with the user based on the interaction object may further include the step of receiving a touch action related to the original number from the user; and the step of detecting a correct answer input event corresponding to the original number based on the touch action.

[0012] In addition, the step of detecting interaction with the user based on the interaction object may further include the step of receiving a stroke related to the original number from the user; and the step of detecting a correct input event corresponding to the original number based on the angle and length of the stroke.

[0013] In addition, based on the interaction object, the step of detecting the interaction with the user may further include the step of receiving an input of a motion from the user to fill a specific area related to the original number; and the step of detecting a correct answer input event corresponding to the original number based on the motion filling more than a certain portion of the specific area.

[0014] In addition, the step of detecting interaction with the user based on the interaction object may further include the step of receiving a line from the user through a pencil, the line connecting to a specific area related to the original number; and the step of detecting a drawing event based on the line leaving the specific area.

[0015] In addition, the step of detecting interaction with the user based on the interaction object may further include the step of receiving a drag motion input from the user; and the step of detecting a movement event based on whether the current page is displayed in an enlarged manner.

[0016] In addition, image files corresponding to each page of the reference book can be received from the server, and area information and question information of the current page displayed to the user can be received from the server.

[0017] Another aspect of the present specification is a user terminal for interacting with a user in a multi-layer structure, comprising: a communication module; a memory; and a processor functionally controlling the communication module and the memory; wherein the processor receives, through the communication module, image files corresponding to each page of a reference book from a server, and generates the multi-layer structure corresponding to the image files, and the multi-layer structure includes 1) a contents layer, 2) an interaction layer, and 3) a canvas layer; and receives, through the communication module, area information and question information of a current page displayed to the user from the server, and based on the area information, generates an interaction object for interaction with the user in the interaction layer corresponding to the current page.

[0018] According to an embodiment of the present specification, interaction with a user can be efficiently achieved in a multi-layer structure for providing the contents of a reference book.

[0019] In addition, according to the embodiment of the present specification, an issue due to overlap between the Interaction Layer and the Canvas Layer of a multi-layer structure can be resolved.

[0020] The effects that can be obtained from this specification are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the technical field to which this specification belongs from the description below.

[0021] Figure 1 is a block diagram illustrating an electronic device related to the present specification.

[0022] FIG. 2 is a block diagram of an AI device according to one embodiment of the present specification.

[0023] Figure 3 is an example of a PDF generation method that can be applied to this specification.

[0024] Figures 4 and 5 are examples of conversion of PDF files to HTML format to which the present specification can be applied.

[0025] Figure 6 shows an application method of the Semantic Box model to which this specification can be applied.

[0026] Figure 7 is an example of a declaration of interaction with a user to which this specification can be applied.

[0027] Figure 8 is an example of Block OCR to which the present specification can be applied.

[0028] Figure 9 is an example of annotation extraction to which this specification can be applied.

[0029] Figure 10 is an example of a method of decomposing each item of an explanatory note to which this specification can be applied.

[0030] Figure 11 is an example of a selection of delimiters to which the present specification can be applied.

[0031] Figure 12 is an example of a breakdown of each item in an explanatory note to which this specification can be applied.

[0032] Figure 13 is an example of data binding to which this specification can be applied.

[0033] Figure 14 is an example of a server to which the present specification can be applied.

[0034] Figure 15 is an example of semantic HTML to which this specification can be applied.

[0035] Figure 16 illustrates a multi-layer structure to which the present specification can be applied.

[0036] Figure 17 illustrates a multi-layer problem to which the present specification can be applied.

[0037] Figure 18 is an example of a method for solving layer overlapping problems using Panel.

[0038] Figure 19 is an example of a terminal to which the present specification can be applied.

[0039] Figure 20 is an example of an event detection method to which the present specification can be applied.

[0040] Figure 21 is an example of a service to which this specification can be applied.

[0041] Figure 22 illustrates a first correct answer input event to which the present specification can be applied.

[0042] Figure 23 illustrates a second correct answer input event to which the present specification may be applied.

[0043] Figure 24 illustrates a third correct answer input event to which the present specification may be applied.

[0044] Figure 25 illustrates drawing events to which the present specification can be applied.

[0045] Figure 26 is an example of a movement event to which this specification can be applied.

[0046] Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of this specification.

[0047] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0048] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0049] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0050] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0051] Figure 1 is a block diagram illustrating an electronic device related to the present specification.

[0052] The above electronic device (100) may include a wireless communication unit (110), an input unit (120), a sensing unit (140), an output unit (150), an interface unit (160), a memory (170), a control unit (180), and a power supply unit (190). The components illustrated in FIG. 1 are not essential for implementing the electronic device, and thus the electronic device described in this specification may have more or fewer components than the components listed above.

[0053] More specifically, among the above components, the wireless communication unit (110) may include one or more modules that enable wireless communication between the electronic device (100) and a wireless communication system, between the electronic device (100) and another electronic device (100), or between the electronic device (100) and an external server. In addition, the wireless communication unit (110) may include one or more modules that connect the electronic device (100) to one or more networks.

[0054] This wireless communication unit (110) may include at least one of a broadcast reception module (111), a mobile communication module (112), a wireless Internet module (113), a short-range communication module (114), and a location information module (115).

[0055] The input unit (120) may include a camera (121) or a video input unit for inputting a video signal, a microphone (122) or an audio input unit for inputting an audio signal, and a user input unit (123, for example, a touch key, a mechanical key, etc.) for receiving information from a user. Voice data or image data collected by the input unit (120) may be analyzed and processed into a user's control command.

[0056] The sensing unit (140) may include one or more sensors for sensing at least one of information within the electronic device, information about the surrounding environment surrounding the electronic device, and user information. For example, the sensing unit (140) may include at least one of a proximity sensor (141), an illumination sensor (142), a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor (G-sensor), a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor (IR sensor), a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor (e.g., a camera (see 121)), a microphone (see 122), a battery gauge, an environmental sensor (e.g., a barometer, a hygrometer, a thermometer, a radiation detection sensor, a heat detection sensor, a gas detection sensor, etc.), and a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric recognition sensor, etc.). Meanwhile, the electronic device disclosed in this specification can utilize information sensed by at least two of these sensors in combination.

[0057] The output unit (150) is for generating output related to visual, auditory, or tactile sensations, and may include at least one of a display unit (151), an audio output unit (152), a haptic module (153), and an optical output unit (154). The display unit (151) may be formed as a layer structure with a touch sensor or formed as an integral part, thereby implementing a touch screen. This touch screen may function as a user input unit (123) that provides an input interface between the electronic device (100) and the user, and at the same time, provide an output interface between the electronic device (100) and the user.

[0058] The interface unit (160) serves as a passageway for various types of external devices connected to the electronic device (100). This interface unit (160) may include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, and an earphone port. In the electronic device (100), appropriate control related to the connected external device can be performed in response to the external device being connected to the interface unit (160).

[0059] In addition, the memory (170) stores data that supports various functions of the electronic device (100). The memory (170) can store a plurality of application programs (or applications) that run on the electronic device (100), data for the operation of the electronic device (100), and commands. At least some of these application programs can be downloaded from an external server via wireless communication. In addition, at least some of these application programs can exist on the electronic device (100) from the time of shipment for the basic functions of the electronic device (100) (e.g., call receiving and sending functions, message receiving and sending functions). Meanwhile, the application programs can be stored in the memory (170), installed on the electronic device (100), and driven by the control unit (180) to perform the operations (or functions) of the electronic device.

[0060] In addition to the operations related to the above application program, the control unit (180) typically controls the overall operation of the electronic device (100). The control unit (180) can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above or by operating an application program stored in the memory (170).

[0061] In addition, the control unit (180) can control at least some of the components discussed with reference to FIG. 1 to drive an application program stored in the memory (170). Furthermore, the control unit (180) can operate at least two or more of the components included in the electronic device (100) in combination to drive the application program.

[0062] The power supply unit (190) receives external power and internal power under the control of the control unit (180) and supplies power to each component included in the electronic device (100). The power supply unit (190) includes a battery, and the battery may be a built-in battery or a replaceable battery.

[0063] At least some of the above components may cooperate with each other to implement the operation, control, or control method of the electronic device according to the various embodiments described below. In addition, the operation, control, or control method of the electronic device may be implemented on the electronic device by driving at least one application program stored in the memory (170).

[0064] In this specification, the electronic device (100) may be collectively referred to as a server, and the server may include a cloud server. In addition, the terminal may include all or part of the electronic device (100) and may include a tablet PC.

[0065] FIG. 2 is a block diagram of an AI device according to one embodiment of the present specification.

[0066] The AI ​​device (20) may include an electronic device including an AI module capable of performing AI processing, or a terminal including the AI ​​module. In addition, the AI ​​device (20) may be included as a component of at least a portion of the electronic device (100) illustrated in FIG. 1 and may be configured to perform at least a portion of the AI ​​processing.

[0067] The above AI device (20) may include an AI processor (21), a memory (25) and / or a communication unit (27).

[0068] The above AI device (20) is a computing device capable of learning a neural network, and can be implemented as various electronic devices such as a terminal, desktop PC, notebook PC, tablet PC, etc.

[0069] The AI ​​processor (21) can learn a neural network using a program stored in the memory (25). In particular, the AI ​​processor (21) can create an AI model for various image classification tasks, such as detecting problem areas in an image, circle numbers, and decomposing problem areas. In addition, the AI ​​model can include a Semantic Box Model for generating Semantic HTML.

[0070] Meanwhile, the AI ​​processor (21) that performs the function described above may be a general-purpose processor (e.g., CPU), but may be an AI-only processor for artificial intelligence learning (e.g., GPU, graphics processing unit).

[0071] The memory (25) can store various programs and data required for the operation of the AI ​​device (20). The memory (25) can be implemented as a non-volatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), a solid state drive (SDD), etc. The memory (25) is accessed by the AI ​​processor (21), and data reading / recording / modifying / deleting / updating, etc. can be performed by the AI ​​processor (21). In addition, the memory (25) can store a neural network model (e.g., a deep learning model) generated through a learning algorithm for data classification / recognition according to one embodiment of the present specification.

[0072] Meanwhile, the AI ​​processor (21) may include a data learning unit that trains a neural network for data classification / recognition. For example, the data learning unit may acquire learning data to be used for learning and apply the acquired learning data to a deep learning model, thereby training the deep learning model.

[0073] The communication unit (27) can transmit the AI ​​processing result by the AI ​​processor (21) to an external electronic device.

[0074] Here, external electronic devices may include other terminals, terminals.

[0075] Meanwhile, the AI ​​device (20) illustrated in FIG. 2 is functionally divided into an AI processor (21), a memory (25), a communication unit (27), etc., but the aforementioned components may be integrated into one module and referred to as an AI module or an artificial intelligence (AI) model.

[0076] Figure 3 is an example of a PDF generation method that can be applied to this specification.

[0077] Referring to Figure 3, PDFs can be categorized as Scanned or Structured depending on the creation method. Scanned PDF (3010) may refer to a file that has been scanned into an image format using a scanner of a specific book, and Structured PDF (3020) may refer to a file that has been electronically created as a PDF using computer software.

[0078] The main difference between Scanned PDF (3010) and Structured PDF (3020) is whether individual elements (e.g. Text, Image, Table) on the page can be dragged from the computer.

[0079] In more detail, Structured PDF (3020) implements all elements on each page in a recognizable form, so when converted to HTML format, a class is assigned to each or all elements individually, enabling implementation of specific class javascript events. However, in the case of Scanned PDF (3010), since the entire page's image is converted into a single element, work must first be done to discover the parts of each image that require interaction and implement the points that require javascript event implementation as overlays.

[0080] Figures 4 and 5 are examples of conversion of PDF files to HTML format to which the present specification can be applied.

[0081] Referring to FIGS. 4 and 5, for example, the terminal can extract elements including an embedded text object, a page full image file, HTML, CSS, Font, and a background vector image from the structured PDF (3020) to convert the structured PDF (3020) into an HTML page.

[0082] In addition, the terminal can use OCR to convert the Scanned PDF (3010) into an HTML page, restore the structure of the Scanned PDF (3010) into a form from which characters, etc. can be extracted, and then extract elements including embedded text objects, Page Full image files, HTML, CSS, Fonts, and Background vector images.

[0083] Afterwards, the terminal can assign a class id value to the identified element in each HTML page.

[0084] Referring to FIG. 5, the terminal can convert the PDF file of FIG. 4 into HTML format.

[0085] However, the class id value of the div included in this converted HTML is generated based on the location information of the top, bottom, left, and right within the PDF page during the process of converting PDF to HTML format, and is not generated based on the area information of the question and the information about the components within the question, making it difficult to use the result immediately in an actual digital service.

[0086] In more detail, since these class ids are not semantically assigned id values ​​based on the problem domain, it is difficult for the terminal to access each problem unit through a specific value.

[0087] In this specification, the terminal can perform the procedure described below to group these location value ids into semantic units (e.g., page / issue / view).

[0088] Figure 6 shows an application method of the Semantic Box model to which this specification can be applied.

[0089] Referring to FIG. 6, the terminal can input an image file of a problem area included in each page of a PDF file into the Semantic Box model.

[0090] The terminal generates a bounding box (6000) surrounding the problem area in each page image using the Semantic Box model. The bounding box can be a rectangle indicating the location and boundaries of an object. The Semantic Box model can extract the location of an object by learning the characteristics of each region contained in the PDF image. The extracted bounding box can then indicate the object's location information.

[0091] For example, the terminal can use OpenCV to determine whether other problem areas exist by dilating adjacent pixels in the remaining area excluding the bounding box (6000).

[0092] Additionally, the terminal can determine that the layout of the page is two columns if the values ​​of the leftmost point x-coordinate (x1) and the rightmost point x-coordinate (x2) of the bounding box (6000) are greater than the midpoint x-coordinate (x half) of the horizontal axis of the page.

[0093] The terminal can extract an area (6100) requiring interaction with the user based on the bounding box (6000). For example, the terminal can first extract a selection area (6010) from the bounding box (6000), and then extract an area (6100) requiring interaction with the user included in the selection area (6010).

[0094] Through this, the terminal can declare an area (6100) where interaction with the user is required, and declare an interaction event area (6300) in the area (6100) where interaction with the user is required.

[0095] For example, the terminal may declare an area (6100) requiring user interaction by detecting a number in a selection area (6010). Thereafter, the terminal may declare an interaction event area (6300) for each number, where the user can select the number.

[0096] The interactive area (6100) in a reference book available on a digital device, where users can input answers, can be used for purposes such as assessing or confirming learner knowledge. This allows learners to assess their level of knowledge, and instructors and educational institutions can track and manage learner achievement.

[0097] Additionally, through the interactive area (6100), learners can engage in hands-on learning by solving problems and entering answers directly from the reference book. This can help increase learner study efficiency.

[0098] That is, the area requiring interaction (6100) is a function necessary for evaluating and confirming learner knowledge and improving learning efficiency.

[0099] Additionally, the terminal can generate a scoring event (6400) based on the learner's answer entered in the area (6100) where interaction is required.

[0100] More specifically, the area requiring user interaction (6100) may include an area containing choice elements for the user's answer, depending on the question in question. Based on the user's answer, the terminal may generate a scoring event (6400) to indicate whether the question is correct or incorrect.

[0101] As an example, if a series of consecutive numbers or parenthetical numbers are detected at the leftmost side of a row and consecutive numbers or parenthetical numbers that increase downward in the column of the numbers or parenthetical numbers are detected, the terminal may extract an area containing sentences containing numbers or parenthetical numbers as an area (6100) requiring interaction with the user.

[0102] The Semantic Box model can be trained to perform the aforementioned actions, and the terminal can obtain location information (e.g., Location, Size, etc.) for the bounding box (6000) and the area (6100) where interaction with the user is required.

[0103] For example, since the extracted bounding box represents a relative position within the image, it may need to be converted to absolute position information as needed. For example, the terminal can calculate the coordinates of the bounding box based on the upper left corner of the image as (0, 0).

[0104] Afterwards, the terminal can combine the non-semantic HTML file generated through the process of converting PDF to HTML with the above location information to wrap the Div included in each area into a semantic meaning unit.

[0105] Alternatively, the above-described procedure may be performed alone, such that the terminal can generate objects associated with the interaction event area (6300) in the image-based file of the PDF.

[0106] Figure 7 is an example of a declaration of interaction with a user to which this specification can be applied.

[0107] Referring to Figure 7, the terminal can declare the problem types of the reference book as multiple choice, short answer (7000), subjective (descriptive), line-by-line (7010), fill-in-the-blank (7020), and O / X (7030).

[0108] The terminal can determine the type of question contained in the problem domain based on the declared problem type. For example, the Semantic Box model can be trained to determine the declared problem type.

[0109] The terminal can declare where the answer is entered and the interaction with the user based on the type of question.

[0110] Table 1 provides examples of where answers are entered and user interaction declarations to which this specification may apply.

[0111] Declaration of distinction Multiple choice Set an event for a single number, or implement multiple choice answer choices in a panel format Short answer Set an answer input format within the question area (e.g., text format) Subjective (descriptive) Set an answer input format within the question area (e.g., text format) Line connection Set input at a specific point so that specific rows can be connected by column, and implement interaction by matching between the two points Fill in the blank Set a location to input text data in parentheses ( ), for example, input format is Text format O / X Set a location to input text data in parentheses ( ), for example, input format is Boolean format

[0112] The terminal can set the interaction objects and / or interaction points described below with reference to Table 1.

[0113] Figure 8 is an example of Block OCR to which the present specification can be applied.

[0114] Referring to FIG. 8, the terminal can group text string data extracted from each image file of a PDF file into adjacent pixels based on semantic units using Block OCR technology.

[0115] Block OCR is a type of OCR technology specifically designed to recognize text blocks. A text block is a chunk of text within a specific section or area of ​​a document, typically with a fixed location and layout within the document.

[0116] For example, Block OCR can first recognize the location and size of these text blocks, and then recognize the text within them. This allows the terminal to remove unnecessary areas or noise, thereby improving character recognition accuracy within the text block.

[0117] More specifically, the terminal can use vision-based geometric information from a given image file to merge adjacent texts and generate text. For example, the terminal can use the Semantic Box model to group texts by semantic unit and infer them as box shapes.

[0118] For example, a terminal can extract meaningful keywords from concept explanation pages within a reference book through Block OCR. This allows the terminal to establish relationships between keywords. Blocks extracted from a page are considered semantically higher in ranking if (i) they are larger in size and (ii) they are located at the top.

[0119] For example, "01 The ruling system of an ancient state" is a higher-level concept than "the formation of a state." Furthermore, "the development of prehistoric culture" and "the emergence of classes and the state" are lower-level concepts than "the formation of a state."

[0120] Figure 9 is an example of annotation extraction to which this specification can be applied.

[0121] A test page containing reference material is typically structured in a two-column format. Each page can contain 4-8 questions. Furthermore, most questions are multiple-choice, with short-answer and subjective questions being less common. Therefore, the key element on a test page is the question area, which can be organized as [question number + question title + multiple-choice answer]. This format can be similar regardless of subject.

[0122] Referring to FIG. 9, the commentary page may have a two column structure (9010) or a two row structure (9020), or various similar forms.

[0123] Unlike question paper pages, commentary pages have fewer shared geometric features. However, all pages and elements within a single reference book can share the same layout design and element shapes. In particular, each reference book series may frequently use unique fonts, styling, and layouts.

[0124] The terminal can classify the data on the commentary page based on the fonts, styling, and layout that are repeatedly used on the commentary page.

[0125] Figure 10 is an example of a method of decomposing each item of an explanatory note to which this specification can be applied.

[0126] Referring to Figure 10, the terminal merges the entire pages of the commentary based on a dividing line (S1010). For example, if the commentary page has a two-column structure (9010), the terminal may merge the page area to the right of the dividing line with the lower part of the page area to the left of the dividing line, using the central vertical line dividing the left and right areas as the dividing line.

[0127] If the commentary page has a two-row structure (9020), the central horizontal line dividing the upper / lower areas can be used as a dividing line, and the lower page area of ​​the dividing line can be merged to the right side of the upper page.

[0128] Additionally, the terminal can merge pages for each page or subsection of the commentary and proceed with the procedure described below.

[0129] Afterwards, the terminal can extract the explanation area from the merged page similar to the problem area extraction method described above.

[0130] The terminal selects a separator from the merged page (S1020). For example, the separator may include unique fonts, styling, and layouts, depending on the elements of the commentary.

[0131] Figure 11 is an example of a selection of delimiters to which the present specification can be applied.

[0132] Referring to FIG. 11, the terminal can select a first separator (1110) for separating chapters in a merged page, a second separator (1120) for separating answer sets, and a third separator (1130) for separating problem answers.

[0133] Referring again to FIG. 10, the terminal acquires delimiter information from the delimiter area (S1030). For example, the delimiter information may include information such as font, styling, and layout related to the delimiter.

[0134] The terminal breaks down the answer sheet into items based on the delimiter information (S1040).

[0135] Figure 12 is an example of a breakdown of each item in an explanatory note to which this specification can be applied.

[0136] Referring to Figure 12, the terminal can detect other similar first identifiers (1220) based on identifier information obtained from the first identifier (1210). To this end, the terminal can utilize computer vision technology. Based on the detected first identifiers, the terminal can decompose the commentary into chapters.

[0137] In addition, the terminal can detect other delimiters, such as the second delimiter and the third delimiter, based on each delimiter information, and thereby decompose the answer sheet into items (1230).

[0138] Figure 13 is an example of data binding to which this specification can be applied.

[0139] Through the aforementioned procedures, the terminal can convert each question in the problem area into Semantic HTML, use this to store and use related elements for each page / question, and match optimized user interactions according to the type of question.

[0140] Alternatively, the terminal can extract each region from the image-based PDF file through the Semantic Box model without performing the Semantic HTML conversion operation, store the related elements for each region on a page / item basis, and match the optimized user interaction based on the type of the question.

[0141] For example, the terminal can decompose the answer sheet into items, so that the correct answer for each item can be known, and the explanation (e.g., text, image, etc.) that shows how the correct answer for each item was derived can be decomposed into item units.

[0142] Referring to FIG. 13, the terminal can perform data binding to establish a relationship between each item and its corresponding explanation.

[0143] For example, the terminal can check whether the number of questions detected in the problem section of the reference page is the same as the number of questions detected in the explanation section of the answer sheet page. To this end, the terminal can use the answers extracted from the explanation section to detect the number of questions included in each chapter and compare this number with the number of questions in the problem section.

[0144] The terminal can perform sequential mapping if the number of questions in the problem area and the number of questions in the explanation area are the same. More specifically, the terminal can establish a relationship between the ID of the problem area and the ID of the explanation area.

[0145] For example, the terminal can group a specific problem area and an explanation area mapped thereto with an ID having a specific value based on location information about the extracted bounding box (6000) and the area requiring interaction with the user (6100).

[0146] Through this, the terminal can receive correct answer information corresponding to each question from the user, match the answer corresponding to each question, and implement automatic scoring and explanation linking functions.

[0147] Additionally, the terminal can bind data extracted from the problem area and explanation area in response to the bundled fingerprint and input metadata information thereon.

[0148] For example, reference book metadata records various attributes of each item, including the item's subject, assessment criteria, and difficulty level. Through this, the terminal can improve the searchability of each item.

[0149] Additionally, the terminal can efficiently manage information in reference books, receive accurate information from users, and efficiently use it for data analysis and utilization.

[0150] Figure 14 is an example of a terminal to which this specification can be applied.

[0151] Referring to Figure 14, the terminal can input a PDF of a reference book and generate an HTML file for each page. For example, the HTML file for each page of the reference book may be in a grid-like HTML format based on class information.

[0152] More specifically, these HTML forms may include classes generated based on a grid. For example, a terminal may obtain an HTML form containing tags based on the positional values ​​of each element, as illustrated in FIG. 4 and / or FIG. 5 described above.

[0153] However, this form may have the problem that it is difficult to implement events such as scoring / answer linking because it does not group semantically, although it displays the same screen as the screen that the user sees on the web page.

[0154] Therefore, in this specification, the terminal can extract bounding box information from an image converted from a PDF file of a reference book and combine it with a grid-type HTML file to generate a semantic HTML file.

[0155] Alternatively, the terminal can input a PDF of a reference book, extract bounding box information, and use this to generate a semantic HTML file based on the image.

[0156] The terminal generates a bounding box related to the problem area based on an image file generated from the reference PDF file (S1410). For example, the problem area may include a question containing the question and answer portions of the question.

[0157] Based on the bounding box, the terminal extracts an area requiring user interaction (S1420). For example, the area requiring user interaction may include elements that allow the user to perform actions to answer questions in the problem area. Additionally, the terminal can obtain location information regarding the bounding box and the area requiring user interaction.

[0158] The terminal generates a semantic HTML file based on the area requiring interaction (S1430). For example, the semantic HTML file may include identifiers that can indicate the content of each element. Elements may include a question number, question text, an image, a sub-question area, a sub-area, and a raw number.

[0159] More specifically, the terminal can determine the type of question contained in the problem area based on the declared question type. Based on the question type, the terminal can declare the location where the answer is entered and / or the interaction with the user.

[0160] The terminal can classify each element within the problem area based on the area requiring interaction. More specifically, in a semantic HTML file, the IDs (e.g., Class IDs) that represent the content (e.g., div elements) within the area requiring interaction can indicate meaning related to the problem area and / or user interaction.

[0161] Figure 15 is an example of semantic HTML to which this specification can be applied.

[0162] Referring to Figure 15(a), the class IDs representing content in areas requiring interaction in a semantic HTML file can indicate meaning related to user interaction. For example, the class ID of the first div may indicate the answer area, the class ID of the second div may indicate the answer box, and the class ID of the third div may indicate a selection item.

[0163] The terminal can generate semantic HTML based on classified elements. For example, based on location information, the terminal can wrap information related to each region (e.g., an area requiring interaction, a problem area) related to the bounding box in an HTML form containing tags based on the positional values ​​of each element. This can be done by creating semantic HTML.

[0164] In addition, referring to FIG. 15(b), the semantic HTML file includes a quiz id for identifying a problem area, and the area information related to the identifier is generated in a data searchable format such as JSON and can be stored in a database.

[0165] Through the above-described procedure, the terminal can identify problem areas contained in the PDF and link corresponding explanations.

[0166] Figure 16 illustrates a multi-layer structure to which the present specification can be applied.

[0167] In the past, reference books were printed on paper, requiring users to use paper and pencil to solve problems. However, recently, reference book content has been provided using electronic devices such as tablet PCs. For example, with a tablet PC, reference book content is presented digitally, allowing users to solve problems using a touch screen or stylus. This technology allows for more efficient and convenient use of reference books, and is widely used in education.

[0168] Referring to Figure 16, tablet PC applications can generally be used in a form where the following three types of layers are stacked:

[0169] 1) Content Layer: This layer visually presents the content provided by the app to the user. This layer can include media formats such as text, images, and more.

[0170] 2) Interaction Layer: This layer enables user interaction. It includes functions for handling events such as touch, swipe, and drag.

[0171] 3) Canvas Layer: This layer is where you can draw or work with the Pencil device. This layer is mainly used in drawing apps, sketching apps, and design apps.

[0172] These layers can be used in an interactive, overlapping manner, each performing its own role. For example, if an image is displayed in the Contents Layer, the user can zoom in or out through the Interaction Layer, while the Canvas Layer allows the user to edit or draw on the image using a Pencil device.

[0173] Figure 17 illustrates a multi-layer problem to which the present specification can be applied.

[0174] Referring to Figure 17, in the multi-layer described above, a problem may arise in which communication between the Canvas Layer and the Interaction Layer is very limited.

[0175] Typically, tablet PC applications structure their applications with the Canvas layer at the top, the Content layer at the bottom, and the Interaction layer somewhere in between. Therefore, events generated using a pencil and / or finger can be attributed to events originating in the top layer.

[0176] For example, since all events originating from the Pencil are assigned first to the Canvas layer, a tablet PC application may be restricted from implementing Interaction events. This can lead to issues due to overlap between the Interaction Layer and the Canvas Layer.

[0177] For example, the Interaction Layer detects a button press event and operates according to the event. However, if a Pencil is used in the Canvas Layer, the Interaction Layer will not detect the button press event and will instead process the Pencil event in the Canvas Layer. In this case, unexpected behavior may occur due to the Pencil event even if the user does not press the button. In other words, the terminal may have a problem in that it is difficult to determine in advance whether a specific event should be attributed to the Canvas layer or the Interaction layer due to the overlapping between layers.

[0178] To address these issues, it's necessary to coordinate communication between the Interaction Layer and the Canvas Layer. For example, when detecting a button press event in the Interaction Layer, the Canvas Layer can be configured to ignore Apple Pencil events. Alternatively, even if the Canvas Layer handles Apple Pencil events, the Interaction Layer can prioritize button press events.

[0179] Typically, this layer overlapping problem can be avoided by implementing separate areas in a Panel format (e.g., writing area, storage area).

[0180] Figure 18 is an example of a method for solving layer overlapping problems using Panel.

[0181] Referring to Figure 18, developers can solve the layer overlapping problem by implementing the Canvas Layer and Interaction Layer as separate Panels without overlapping them.

[0182] However, this method may have several problems.

[0183] First, while there may be no problem when the user inputs simple answers, such as text or numbers, entering the answers can be challenging in other cases. For example, in a problem requiring graph drawing, the user must use a stylus to draw on the Canvas Layer. However, if the Interaction Layer is implemented as a separate panel, it's difficult to draw freely, as with paper and pencil, which can impact accuracy.

[0184] Second, in the case of the method of Fig. 18, there is a fatal flaw in that the layout of a reference book in the form of a book cannot be utilized because only one item can be displayed on the display.

[0185] Third, solving problems in a reference book must provide users with a learning experience that goes beyond simply solving problems. Therefore, the application needs to provide this experience as realistically as possible. However, implementing it as a separate panel differs from reality, making it difficult to realistically provide this experience to users.

[0186] Figure 19 is an example of a terminal to which the present specification can be applied.

[0187] Referring to Figure 19, the terminal may include a tablet PC, and the terminal may be connected to a server to transmit and receive data. The terminal may be equipped with a reference book-related application, providing users with a service for solving reference book problems.

[0188] The terminal can implement a multi-layer structure, which can include 1) a content layer, 2) an interaction layer, and 3) a canvas layer. The terminal can display image files of a reference book through the content layer.

[0189] The terminal sets an interaction point in the generated semantic HTML file or the image file of the reference book through the procedure of FIG. 14 (S1910).

[0190] For example, interaction points can be established based on the location values ​​for areas requiring interaction contained in a semantic HTML file. Alternatively, each element contained in an image file of a reference book can be classified and established based on the location values ​​for elements requiring interaction. These interaction points can be included in specific areas described below. For example, a specific area can include an area requiring interaction. Therefore, in this specification, "interaction points" can be replaced with "specific areas."

[0191] The terminal creates an interaction object at the interaction point (S1920).

[0192] For example, the terminal can create an interaction object of the form illustrated in FIG. 15 based on an image file of a reference book.

[0193] Here, an interaction object can refer to an object that a user can interact with by touching a specific area or using a stylus or mouse. For example, in a math problem-solving application, the answer input field for the problem could be an interaction object. The user can enter numbers or formulas into this field using a stylus, and the terminal can recognize this and perform appropriate processing.

[0194] Additionally, a button that displays an explanation of a problem can also be an interactive object. When the user touches this button, the explanation for the problem appears, and when the user touches it again, the explanation disappears.

[0195] For example, interaction objects are identified by item and assigned an ID value, which can be associated with the explanation of that item.

[0196] The terminal can create an interaction object based on the interaction declaration in Table 1 described above. The terminal can transparently create the interaction object and display it to the user.

[0197] More specifically, the terminal can create a transparent layer (e.g., an interaction layer) over an image file that can create interactive objects. The terminal can set a specific area in the interaction layer and create an interactive object (e.g., a button input object) to be included in the specific area.

[0198] The terminal detects an interaction based on the interaction object (S1930).

[0199] An interaction may be an interaction action between a terminal and a user. For example, an interaction may include a user performing an interaction action related to an interactive object on a Canvas Layer using a finger and / or a Pencil.

[0200] The terminal determines whether an event related to the specific area has occurred based on the interaction detected (S1940).

[0201] For example, events related to a specific area may include an answer entry event in which a user enters a correct answer, a drawing event in which a user draws a picture, and / or a movement event in which a user requests a screen movement between pages or within a page.

[0202] Figure 20 is an example of an event detection method to which the present specification can be applied.

[0203] Referring to Figure 20, the terminal can set a specific area containing points where interaction actions are connected. For example, the terminal can define the specific area using an artificial intelligence model. This specific area may be a range expected to receive interaction actions from the user.

[0204] More specifically, within the interaction layer, a "specific region" can refer to a designated area for recognizing user input. These regions are typically expressed in screen coordinates and can be used to detect user interactions, such as touches or clicks. For example, elements displayed on the screen, such as buttons, links, or circular symbols, each correspond to a specific region, and when the user selects that region, the terminal can detect and process the corresponding interaction.

[0205] The terminal determines whether the interaction occurred and was completed within a specific area (S2010).

[0206] The terminal detects a first correct input event based on the interaction being a touch action when the interaction occurs and is completed within a specific area (S2100). For example, the terminal can recognize a short touch interaction lasting less than 100 ms as a touch action. In this case, the terminal can detect a correct input event and determine that there is no drawing event.

[0207] Figure 22 illustrates a first correct answer input event to which the present specification can be applied.

[0208] Referring to FIG. 22, the terminal can receive a touch gesture input from the user. For example, the terminal can create a button input object corresponding to the original numbers as an interaction object, and the interaction object can be included in a specific area (2200).

[0209] When a touch action (2210) input by a user to a terminal occurs and is completed within a specific area (2200) (for example, when a touch using a user's finger (or pencil) is made within a specific area and the separation of the finger (or pencil) is completed within the specific area), the terminal can detect a first correct answer input event.

[0210] When a correct answer entry event is detected, the terminal may display a check mark over the corresponding number to indicate that a correct answer has been entered.

[0211] Referring again to FIG. 20, the terminal detects a second correct answer input event based on the angle and length of the stroke, based on the interaction being a motion other than a touch motion (S2200).

[0212] For example, if the angle of the stroke is between 60 and 120 degrees and the length is shorter than a specific length, the terminal can detect a correct input event.

[0213] Figure 23 illustrates a second correct answer input event to which the present specification may be applied.

[0214] Referring to FIG. 23, the terminal may receive input from the user of a motion other than a touch motion (e.g., a touch of 100 ms or longer). For example, the terminal may create a button input object corresponding to the original numbers as an interaction object, and the interaction object may be included in a specific area (2300).

[0215] A user can input a motion other than a touch motion via a pencil. If a motion other than a touch motion (2310) input by the terminal from the user occurs and is completed within a specific area (2300) (e.g., a touch via a pencil is made within a specific area and a drawn stroke is completed within the specific area), the terminal can determine the angle and length of the stroke, and if the angle and length are within a specific range, can detect a second correct answer input event.

[0216] Referring again to FIG. 20, the terminal detects a third correct input event if the interaction fills more than a certain portion of a specific area based on the interaction being a motion other than a touch motion (S2300). For example, if the form of the interaction-related motion does not have a corner but fills more than [60%] of the specific area, the terminal may detect a correct input event and determine that there is no drawing event.

[0217] Figure 24 illustrates a third correct answer input event to which the present specification may be applied.

[0218] Referring to FIG. 24, if the terminal determines that a form of a motion other than a touch motion input from a user does not include a stroke but fills a certain portion or more of a specific area (2400), the terminal can detect a third correct answer input event.

[0219] Referring again to FIG. 20, the terminal detects a drawing event based on whether the interaction occurs within a specific area and is not completed within the specific area, whether the interaction is input via a pencil, and whether the interaction is associated with the specific area (S2001). For example, the case where the interaction is associated with the specific area may include: 1) the interaction occurs outside the specific area and is completed within the specific area; 2) the interaction occurs outside the specific area, passes through the specific area, and is completed outside the specific area; and / or 3) the interaction occurs within the specific area and is completed outside the specific area.

[0220] Figure 25 illustrates drawing events to which the present specification can be applied.

[0221] Referring to FIG. 25, the terminal can receive interaction input from the user through a pencil, and the interaction can be linked to a specific area.

[0222] For example, a user may draw a line with a pencil starting from outside a specific area, passing through the specific area, and ending outside the specific area (2510).

[0223] Alternatively, the user may draw a line with a pencil starting from outside a specific area and ending within a specific area (2520).

[0224] Referring again to FIG. 20, if the interaction is input via a tool other than a pencil (e.g., a finger) and is a drag motion, the terminal detects a movement event (S2002). For example, if the page on which the interaction is input maintains its original aspect ratio, the terminal may detect the drag motion as a page-to-page movement event (e.g., a swipe). Alternatively, if the page on which the interaction is input is enlarged beyond its original aspect ratio, the terminal may detect the drag motion as a page-to-page movement event.

[0225] Figure 26 is an example of a movement event to which this specification can be applied.

[0226] Referring to FIG. 26, a user can input a drag motion to the terminal using a finger. The terminal can detect a movement event when a drag motion is input using a tool other than a pencil.

[0227] If the current page is not enlarged, the terminal can detect the drag action as a page-to-page movement event (2610).

[0228] If the current page is zoomed in, the terminal can detect the drag action as a movement event within the page (2620).

[0229] Referring again to Figure 19, the terminal transmits the event related to a specific area to the Interaction Layer for processing (S1950).

[0230] For example, based on the interaction object, the terminal can transmit the corresponding event to the Interaction Layer to perform an appropriate action. More specifically, in the case of a correct answer input event, the terminal can receive the correct answer from the user, check the correct answer (2320), and display it to the user. In the case of a drawing event, the terminal can display a drawing effect to the user.

[0231] If the event is not related to a specific area, the terminal can forward the event to the Cnavas Layer for processing.

[0232] Through this, the user can solve problems using a stylus, etc., and the terminal can help the user solve the problem through appropriate interaction between the Interaction Layer and Canvas Layer.

[0233] Additionally, the terminal transmits the information required by the user to the management server during the problem-solving process, and the management server can store or analyze the information and utilize it.

[0234] Figure 21 is an example of a service to which this specification can be applied.

[0235] Referring to Figure 21, an embodiment of a service for solving a user's reference book problem is illustrated.

[0236] A user can utilize a service for solving reference book problems related to the embodiments of FIGS. 19 and 20 by connecting to a first terminal (server, e.g., a cloud server) via a second terminal (user terminal, e.g., a tablet PC). To this end, a separate application providing a service for solving reference book problems may be installed on the second terminal. Furthermore, the first terminal and / or the second terminal may perform all or part of the functions of the terminals described herein.

[0237] The second terminal transmits an image file of the reference book to the first terminal (S2110). For example, the image file of the reference book may include a PDF file containing the contents of each page within the reference book. Thereafter, the first terminal may extract bounding box information from the image file. For example, the first terminal may detect a problem area from each page of the image file, extract an area requiring interaction (e.g., a choice area) within the problem area, and detect each element (e.g., a choice element) within the area requiring interaction.

[0238] For example, the first terminal can store area information related to the extracted bounding box information in a DB (Data Base) in the form of Figure 15(b).

[0239] Additionally, the first terminal can detect the explanation area of ​​the reference book's image file and extract the answer number and correct answer elements from the explanation area. For example, the first terminal can decompose the explanation page of the reference book into question-by-question sections and decompose the explanation elements for each question into question-by-question sections. Through this, the first terminal can extract explanation area information for each question section and, as described above in Figure 13, store the explanation area information in the database by binding it with the area information for related questions.

[0240] Thereafter, the first terminal may generate an image of the question and / or explanation in the corresponding area based on the question area information and / or explanation area information, and store it in file storage. Furthermore, the database and / or file storage may be configured to be accessible only to users of the second terminal.

[0241] The second terminal receives from the user the selection of reference books for which the service is intended for solving reference book problems (S2120). For example, the second terminal may receive from the first terminal a list of reference books available for the service. The list of available reference books may be generated based on the image files of the reference books transmitted by the second terminal to the first terminal.

[0242] For example, a first terminal may generate a list of possible reference books based on the image file of the reference book received from a second terminal. To this end, the database may include domain information of multiple reference books, and the first terminal may identify information about the received reference book (e.g., the title and year of publication of the received reference book) based on the image file of the reference book received from the second terminal.

[0243] More specifically, the first terminal can compare the features of the domain information of multiple reference books included in the DB with the features of the domain information generated from the image file of the reference book received from the second terminal, and identify the information of the received reference book based on similarity. Based on the identification information of the received reference book, the first terminal can generate a list of possible reference books for the second terminal and transmit it to the second terminal, and the second terminal can display the list of possible reference books to the user, thereby allowing the user to select a reference book that is the target of the service.

[0244] The second terminal receives images of each page of the selected reference book from the first terminal (S2130). For example, the second terminal may receive images of the selected reference book's questions and / or explanations contained in the file storage.

[0245] The second terminal creates multiple layers corresponding to the images on each page (S2140). For example, the second terminal can transparently create a canvas layer for receiving user handwriting input and an interaction layer for user interaction. Furthermore, the second terminal can create a content layer for displaying the images on each page.

[0246] Additionally, the second terminal can create an event listener that can detect interaction-related events, taking precedence over all layers, to determine whether an event related to a specific area has occurred.

[0247] The second terminal receives the current page's area information and item information from the first terminal (S2150). For example, the second terminal can receive area information and item information contained in the database.

[0248] The second terminal creates an interaction object in the interaction layer corresponding to the current page based on the domain information and question information (S2160). For example, the second terminal may perform the operations of S1910 and / or 1920 described above based on the image file of the current page. The current page may refer to a page of a reference book that the second terminal is displaying to the user for problem solving.

[0249] Thereafter, the second terminal can detect an interaction based on the interaction object according to the operations of S1930 to S1950 described above. For example, the second terminal can detect an interaction through an event listener. Thereafter, based on the detected interaction, the second terminal can determine whether an event related to the specific area has occurred, and based on the event related to the specific area, the second terminal can forward the event to the Interaction Layer for processing.

[0250] As another example, the second terminal can directly generate bounding box information without transmitting the reference book's image file to the first terminal. For example, the second terminal can directly generate the region and question information generated by the first terminal using a lightweight AI model. Then, the second terminal can create an interaction object in the interaction layer based on the region information it directly generated.

[0251] In this case, the second terminal can process the image directly without transmitting or receiving it to the first terminal. Furthermore, the step of transmitting or receiving image files corresponding to reference pages to the first terminal may not be necessary, nor may the step of receiving generated region information be required. Furthermore, by executing the model on the second terminal to create an interaction layer, the load associated with data transmission can be reduced, enabling real-time service provision to users.

[0252] The above-described specification can be implemented as computer-readable code on a program-recorded medium. A computer-readable medium includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc., and also includes media implemented in the form of carrier waves (e.g., transmission via the Internet). Therefore, the above detailed description should not be construed as limiting in all respects, but rather as illustrative. The scope of this specification should be determined by a reasonable interpretation of the appended claims, and all changes within the equivalent scope of this specification are intended to be included within the scope of this specification.

[0253] In addition, while the above description focuses on services and embodiments, these are merely examples and do not limit the present disclosure. Those skilled in the art to which this disclosure pertains will appreciate that various modifications and applications not exemplified above are possible without departing from the essential characteristics of the present disclosure. For example, each component specifically shown in the embodiments can be modified and implemented. In addition, differences related to such modifications and applications should be construed as being included within the scope of this disclosure as defined in the appended claims.

Claims

1. In a method for a user terminal to interact with a user in a multi-layer structure, A step of generating the multi-layer structure based on image files corresponding to each page of a reference book; wherein the multi-layer structure includes 1) a content layer, 2) an interaction layer, and 3) a canvas layer; and A step of creating an interaction object for interaction with the user in an interaction layer corresponding to the current page based on the area information and question information of the current page displayed to the user; An implementation method including:

2. In paragraph 1, A step of detecting an interaction with the user based on the above interaction object; and A step of transmitting the detected event to the interaction layer and processing it based on the detected interaction; An implementation method including:

3. In paragraph 2, Based on the above interaction object, the step of detecting the interaction with the user A step of receiving a touch action related to a number from the user; and A step of detecting a correct answer input event corresponding to the original number based on the above touch action; An implementation method including further.

4. In paragraph 2, Based on the above interaction object, the step of detecting the interaction with the user A step of receiving a stroke related to a number from the user; and A step of detecting a correct input event corresponding to the original number based on the angle and length of the stroke; An implementation method including further.

5. In paragraph 2, Based on the above interaction object, the step of detecting the interaction with the user A step of receiving an input from the user for filling in a specific area related to the original number; and A step of detecting a correct answer input event corresponding to the original number based on the above action filling a certain portion or more of the specific area; An implementation method including further.

6. In paragraph 2, Based on the above interaction object, the step of detecting the interaction with the user A step of receiving a line connected to a specific area related to a circle number from the user through a pencil; and A step of detecting a drawing event based on the above line leaving the above specific area; An implementation method including further.

7. In paragraph 2, Based on the above interaction object, the step of detecting the interaction with the user A step of receiving a drag motion input from the user; and A step of detecting a movement event based on whether the current page is displayed enlarged; Implementation method including further 8. In paragraph 1, A step of receiving image files corresponding to each page of the reference book from the server; and A step of receiving area information and question information of the current page displayed to the user from the server; An implementation method including:

9. In a user terminal for interacting with a user in a multi-layer structure, Communication module; memory; and A processor that functionally controls the communication module and the memory; The above processor Based on the image files corresponding to each page of the reference book, the multi-layer structure is created, and the multi-layer structure includes 1) a content layer, 2) an interaction layer, and 3) a canvas layer. Based on the area information and question information of the current page displayed to the user, an interaction object for interaction with the user is created in the interaction layer corresponding to the current page. User terminal.

Images