Metaverse English education system using role-playing LLM-based NPCs

Abstract

The present invention relates to an English education system using virtual roleplay, comprising: a metaverse space providing device; a story providing device that generates a story, generates a Non-Player Character (NPC) that intervenes in the story, and generates an interaction scenario between the NPC and a user; and a Head Mounted Display (HMD) device that provides content according to the interaction scenario provided by the story providing device in the space provided by the metaverse space providing device and generates a user input signal corresponding to the content.

Description

Technology Field

[0001] The present invention relates to a metaverse English education system based on a role-playing method utilizing LLM-based NPCs. Background Technology

[0002] The demand for non-face-to-face and contactless education is growing. To meet this demand, online education is being utilized in both the public and private sectors. This online education takes place either by students playing pre-generated educational content on their own devices or by instructors conducting lectures via video while connected simultaneously with students.

[0003] Online education based on conventional technology is inevitably less efficient than offline education due to a lack of concentration. Since mutual interaction is difficult in an online environment, it is necessary to develop an educational system designed to engage students with relatively low attention spans.

[0004] Meanwhile, augmented reality and virtual reality technologies are also steadily advancing. Research on the introduction of these technologies as methods for non-face-to-face and contactless education is becoming increasingly active. However, relying solely on the introduction of these technologies has had limitations in fostering rapport between instructors and students and enhancing educational concentration. Prior art literature

[0005] Published Patent 10-2003-0017850 The problem to be solved

[0006] The present invention aims to provide an English education system that increases the concentration of education using virtual roleplay in order to solve the aforementioned problems.

[0007] The problems of the present invention are not limited to those mentioned above, and other unmentioned problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0008] To achieve the above objective, an English education system using virtual roleplay according to an embodiment of the present invention comprises: a metaverse space providing device; a story providing device that generates a story, generates a Non-Player Character (NPC) that intervenes in the story, and generates an interaction scenario between the NPC and a user; and a Head Mounted Display (HMD) device that provides content according to the interaction scenario provided by the story providing device in the space provided by the metaverse space providing device, and generates a voice input signal corresponding to the content.

[0009] Specific details of other embodiments are included in the detailed description and drawings. Effects of the invention

[0010] According to the present invention, there is one or more of the following effects.

[0011] First, it has the effect of improving students' immersion in non-face-to-face English learning.

[0012] Second, since the story unfolds based on interactions with NPCs, English education can be conducted while immersed in the narrative, which is effective in improving academic achievement.

[0013] Third, it has the effect of purifying the educational environment by filtering abnormal and unethical remarks.

[0014] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description in the claims. Brief explanation of the drawing

[0015] FIG. 1 is a drawing referenced to explain a system according to an embodiment of the present invention. FIG. 2 is a flowchart according to an embodiment of the present invention. FIG. 3 is a drawing referenced to explain the story of a virtual roleplay according to an embodiment of the present invention. FIG. 4 is a drawing referenced to explain the filtering operation according to an embodiment of the present invention. FIG. 5 is a drawing referenced to explain an interaction scenario according to an embodiment of the present invention. Specific details for implementing the invention

[0016] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols will be assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0017] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0018] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0019] A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0020] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0021] FIG. 1 is a drawing referenced to explain a system according to an embodiment of the present invention.

[0022] Referring to the drawing, the system (1) can be named an English education system using virtual roleplay.

[0023] The system (1) can create a metaverse space and conduct education while performing virtual roleplay within the metaverse space. The system (1) can be used for non-face-to-face English education.

[0024] The system (1) may include a video providing device (20), a database (30), a story providing device (40), a metaverse space providing device (100), and a user device (300).

[0025] Each of the video providing device (20), database (30), story providing device (40), metaverse space providing device (100), and user device (300) can communicate via a wired or wireless network. The wired or wireless network may utilize infrastructure provided by a third party.

[0026] The image providing device (20) can generate an image that serves as the background for the implementation of the metaverse space.

[0027] The image providing device (20) may include a 360-degree camera, a stereo camera, and a lidar for generating images.

[0028] The image providing device (20) can generate an image that serves as the background for the implementation of the metaverse space by dividing it into a distant view, a middle view, and a close view.

[0029] Distant, middle, and near views can be described as a classification based on the distance perceived by a user wearing an HMD.

[0030] The distant view can be described as a background image that allows a user wearing an HMD to perceive only the silhouettes of objects. The distant view may be a background image generated to allow a user wearing an HMD to feel a sense of distance of about 2 km to 3 km.

[0031] A distant view can be generated based on an image generated from a 360-degree camera. The image providing device (20) can generate a distant view image based on a 360-degree image included in a 360-degree camera.

[0032] Midground can be described as a background image that enables a user wearing an HMD to perceive the shape of an object. Midground can be a background image generated to allow a user wearing an HMD to feel a sense of distance of about 5m to 2km.

[0033] A midground image can be generated based on an image generated from a stereo camera. The stereo image generated from the stereo camera includes depth information, and the image providing device (20) can generate a midground image with a sense of distance by implementing depth information in each object included in the image.

[0034] The foreground can be described as a background image that allows a user wearing an HMD to clearly perceive objects. The foreground can be a background image generated to allow a user wearing an HMD to feel a sense of distance of about 0m to 5m.

[0035] The foreground can be generated by combining an image generated from a LiDAR with an image generated from a stereo camera.

[0036] Considering the mutually complementary characteristics of the lidar and the stereo camera, accurate depth information can be obtained by focusing the scan on the horizontal direction where the lidar measures a relatively long distance, and supplementing the depth information of the remaining area using data obtained from the stereo camera. The image providing device (20) can generate a near-field image by reflecting more accurate depth information compared to the mid-field image in the image generated by the stereo camera.

[0037] The video providing device (20) can provide the video to the metaverse space providing device (100).

[0038] The database (30) can store data for multiple objects. The database (30) can store data for graphically implementing the objects. The database (30) can store data for various objects related to the story generated by the story providing device (40).

[0039] The database (30) can provide object data to the metaverse space providing device (100) based on a signal received from the metaverse space providing device (100).

[0040] The story providing device (40) can provide the story to the metaverse space providing device (100).

[0041] The story providing device (40) can generate a story using an artificial intelligence learning model.

[0042] The story providing device (40) can generate a story.

[0043] The story providing device (40) can obtain information about a basic story from the manager and expand the basic story using an artificial intelligence model. For example, if the manager provides a basic story with a space theme, the story providing device (40) can expand story elements corresponding to the basic story using an artificial intelligence learning model.

[0044] Here, story elements may include storylines, outlines, synopses, portfolios, etc.

[0045] A storyline is a collective term for the resulting blueprint, structure, and development of a story prior to the content formation stage; an outline is the result of planning and recording the general outline and ideas of the story structure; and a synopsis refers to the result that includes the development of the story structure and the plot (beginning, middle, and end). Furthermore, a portfolio refers to the creative or recorded result that encompasses the overall structure and development process of the story, key sentences for each plot, and major setting information for each chapter.

[0046] The story providing device (40) can generate a story by automatically completing a storyline including an outline, synopsis, portfolio, etc.

[0047] The story providing device (40) can obtain content that constitutes story elements expanded from the manager's basic story using an artificial intelligence model. The artificial intelligence model can generate a first story that includes the manager's basic story.

[0048] The first story can be defined as a story in which story elements are expanded by an artificial intelligence model based on a basic story input by the manager.

[0049] The story providing device (40) can generate NPCs (Non-Player Characters) that intervene in the story.

[0050] The story providing device (40) can generate multiple characters that intervene in the first story.

[0051] In the first story, multiple characters may appear. Among the multiple characters intervening in the first story, characters other than those linked to the HMD device (300) may be described as NPCs.

[0052] The NPC can converse and interact with the user of the HMD device (300). The story providing device (40) can support the NPC's actions using an artificial intelligence model based on an LLM (Large Language Model).

[0053] The story providing device (40) can generate an interaction scenario between an NPC and a user.

[0054] The story providing device (40) can obtain other elements that can modify or expand the first story based on a signal received from the HMD device (300), and can generate a second story based thereon.

[0055] The second story can be defined as a story in which the first story is modified or expanded based on a signal received from the HMD device (300).

[0056] The HMD device (300) can generate a signal based on user input through interaction between the NPC and the user.

[0057] The story providing device (40) can generate an interaction scenario based on a signal received from the HMD device (300). The story providing device (40) can generate an interaction scenario based on a user input signal received from the HMD device (300). For example, the story providing device (40) can generate an interaction scenario by processing a voice input signal into natural language.

[0058] Meanwhile, the interaction scenario can be described as part of a second story that reflects a signal received from the HMD device (300) in the first story.

[0059] The story providing device (40) can generate a story in the form of a musical.

[0060] The story providing device (40) can generate a story in the form of a musical so that English education can be provided through activities including chants, songs, and musical performances.

[0061] The metaverse space providing device (100) can implement an educational metaverse space (MTV). The metaverse space can be defined as a three-dimensional virtual world.

[0062] The metaverse space providing device (100) can implement an educational metaverse space (MTV) based on signals, information, or data received from a video providing device (20), a database (30), a story providing device (40), and a user device (300).

[0063] The metaverse space providing device (100) can receive an image that serves as the background for the implementation of the metaverse space from the image providing device (20). The metaverse space providing device (100) can process the received image to form a metaverse space (MTV).

[0064] The metaverse space providing device (100) can receive graphic object data from a database (30). The metaverse space providing device (100) can create objects within the metaverse space.

[0065] The metaverse space providing device (100) can receive story data from the story providing device (40). The metaverse space providing device (100) can provide an image based on the story within the metaverse space.

[0066] The metaverse space providing device (100) can receive a signal related to avatar movement from a user device (300). Based on the received signal, the metaverse space providing device (100) can create an avatar in the metaverse space and control the movement of the avatar.

[0067] The metaverse space providing device (100) can provide a space divided into a distant view, a middle view, and a near view.

[0068] The metaverse space providing device (100) can implement a space based on a background image received from an image providing device (20). At this time, based on a distant image, a mid-range image, and a near image received from the image providing device (20), it can provide a space divided into distant, mid-range, and near.

[0069] The metaverse space providing device (100) can implement an image corresponding to an NPC in the metaverse space.

[0070] The user device (300) can be used for the user to enter the metaverse space. The user device (300) can be understood as an electronic device that the user can wear.

[0071] The user device (300) can be described as a Head Mounted Display (HMD) device. Meanwhile, the HMD device (300) may utilize disclosed technology.

[0072] The HMD device (300) can stimulate at least one of the senses of sight, hearing, and touch so that the user perceives that they are located in the metaverse space.

[0073] The HMD device (300) can transmit signals generated from the HMD device (300) to the metaverse space providing device (100) and the story providing device (40).

[0074] The HMD device (300) can transmit signals related to an avatar generated in the metaverse space to the HMD device (300).

[0075] The HMD device (300) can provide digital content according to an interaction scenario provided by the story providing device (40) in the space provided by the metaverse space providing device (100).

[0076] The HMD device (300) can generate a user input signal corresponding to the content. Here, the user input signal can be described as a signal generated by a plurality of sensors equipped in the HMD device (300) based on the user's intention. For example, the user input signal may be a signal that converts the user's voice into an electrical signal.

[0077] The HMD device (300) can generate user input signals generated by the interaction between the NPC and the user.

[0078] FIG. 2 is a flowchart according to an embodiment of the present invention.

[0079] Referring to the drawing, the story providing device (40) can be operated using an artificial intelligence model based on an LLM (Large Language Model).

[0080] The story providing device (40) can generate a story (S210).

[0081] The story providing device (40) can obtain content that constitutes story elements expanded from the manager's basic story using an artificial intelligence model. The artificial intelligence model can generate a first story that includes the manager's basic story.

[0082] The story providing device (40) can generate an NPC (Non-Player Character) that intervenes in the story (S220).

[0083] The story providing device (40) can generate multiple characters appearing in the first story. The story providing device (40) can implement multiple characters as NPCs.

[0084] When a signal is received from the HMD device (300), the story providing device (40) may designate one of the multiple characters as a character linked to the HMD device (300). In this case, among the multiple characters intervening in the first story, characters other than the character linked to the HMD device (300) may be described as NPCs.

[0085] The story providing device (40) can receive a voice input signal from the HMD device (300) (S230).

[0086] The voice input signal may be a signal based on voice input generated by interaction with an NPC.

[0087] The story providing device (40) can generate an interaction scenario based on a voice input signal (S240).

[0088] The interaction scenario is a scenario generated by the interaction between the NPC and the user, and can be described as part of a second story that reflects the signal received from the HMD device (300) in the first story.

[0089] FIG. 3 is a drawing referenced to explain the story of a virtual roleplay according to an embodiment of the present invention.

[0090] Referring to the drawing, the story providing device (40) can generate a story that implements steps S310 through S350.

[0091] The metaverse space providing device (100) can provide content based on the story provided by the story providing device (40).

[0092] The metaverse space providing device (100) can provide content that starts a role-playing game and introduces the world of the game (S310).

[0093] The metaverse space providing device (100) can select a first unit in the provided space based on a signal received from the HMD device (300) (S320).

[0094] Meanwhile, units can be described as sub-stories that distinguish stories, and a single story can be divided into multiple units.

[0095] The metaverse space providing device (100) can provide content corresponding to the first unit (S330).

[0096] With the first unit provided, the HMD device (300) can interact with an NPC. Interaction may include performing tasks based on conversation with the NPC and performing chant and song activities.

[0097] With the first unit provided, the metaverse space providing device (100) can evaluate the user based on a signal received from the HMD device (300).

[0098] If the evaluation results satisfy the pre-set criteria, you can move to the next unit.

[0099] When content corresponding to all units included in the first story is provided, the metaverse space providing device (100) can perform the final stage performance (S340).

[0100] If the criteria for performing an evaluation in the final stage performance are satisfied, the metaverse space providing device (100) can proceed to the next level (S350).

[0101] FIG. 4 is a drawing referenced to explain the filtering operation according to an embodiment of the present invention.

[0102] Referring to the drawing, the story providing device (40) can perform filtering.

[0103] The story providing device (40) can receive voice input spoken by the user from the HMD device (300).

[0104] The story providing device (40) can classify voice input, refine it, and output the NPC interaction voice corresponding to the voice input.

[0105] The story providing device (40) may include a speech classifier (420), a speech purification model (430), and a chatbot (440).

[0106] The speech classifier (420) can be implemented as an LLM (Large Language Model).

[0107] The speech classifier (420) can be trained with an open benchmark dataset (e.g., jigsaw).

[0108] The speech classifier (420) can classify voice input received from the HMD device (300) into ethical speech and unethical speech after machine learning.

[0109] The speech classifier (420) can transmit speech classified as unethical speech to the speech purification model (430).

[0110] The speech purification model (430) can be implemented as an LLM.

[0111] The speech purification model (430) can be trained using an open benchmark dataset (e.g., ParaDetox).

[0112] The speech purification model (430) can purify the received unethical speech into similar ethical speech and provide it to the chatbot (440) after machine learning.

[0113] The chatbot (440) can be implemented as an LLM.

[0114] The chatbot (440) can input ethical remarks provided by the speech purification model (430) and provide remarks that interact with the input remarks as the voice of an NPC.

[0115] FIG. 5 is a drawing referenced to explain an interaction scenario according to an embodiment of the present invention.

[0116] Referring to the drawing, the story providing device (40) can generate a first story and provide it to the metaverse space providing device (100).

[0117] The metaverse space providing device (100) can generate a first space (610) corresponding to a first story. The generated space (610) can be represented by an HMD device (300).

[0118] In the first space (610), the user can converse with an NPC in English through the HMD device (300). Depending on the interaction between the user and the NPC, the story providing device (40) can change the story. The metaverse space providing device (100) can create a space (620, 630) corresponding to the changed story. The created space (620, 630) can be represented by the HMD device (300).

[0119] When a first interaction occurs in the first space (610), the story providing device (40) can change the first story to create a second story. The metaverse space providing device (100) can create a second space (620) corresponding to the second story.

[0120] When a second interaction different from the first interaction occurs in the first space (610), the story providing device (40) can change the first story to create a third story different from the second story. The metaverse space providing device (100) can create a third space (630) corresponding to the third story.

[0121] The foregoing invention may be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Additionally, the computer may include a processor or a control unit. Accordingly, the above detailed description should not be interpreted restrictively in all respects and should be considered exemplary. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention. Explanation of the symbols

[0122] 1 : System 40: Story providing device 100 : Metaverse space provider 300 : HMD

Claims

Claim 1 An image providing device including a 360-degree camera, a stereo camera, and a LiDAR; a metaverse space providing device that receives a background image from the image providing device to implement a metaverse space, and provides a space divided into distant, mid-range, and near views; and a story providing device that generates a story, generates a Non-Player Character (NPC) that intervenes in the story, and generates an interaction scenario between the NPC and a user. and a Head Mounted Display (HMD) device that provides content according to an interaction scenario provided by the story providing device in the space provided by the metaverse space providing device and generates a user input signal corresponding to the content; wherein the distant view is generated based on an image generated by the 360-degree camera, the mid-range view is generated based on an image generated by the stereo camera, and the near view is generated by combining an image generated by the LiDAR with an image generated by the stereo camera, wherein the LiDAR is generated by scanning the horizontal direction and supplementing the depth information obtained from the stereo camera with the image obtained; the story providing device generates a first story, which is a story in which story elements are expanded by an artificial intelligence model based on a basic story input by an administrator, and when a first interaction occurs between a user and an NPC via the HMD in the first space generated by the metaverse space providing device, the first story is modified to generate a second story, and when a second interaction occurs between a user and an NPC via the HMD in the first space, the first story is modified to be different from the second story. An English education system using virtual roleplay that generates a third story. Claim 2 In claim 1, the story providing device is an English education system using virtual roleplay that generates the interaction scenario based on the user input signal received from the HMD device. Claim 3 In claim 2, the story providing device is an English education system using virtual roleplay that generates the interaction scenario by processing a voice input signal into natural language. Claim 4 In Clause 3, the story providing device is an English education system using virtual roleplay that generates a story in the form of a musical. Claim 5 In claim 1, the metaverse space providing device is an English education system using virtual roleplay that implements an image corresponding to the NPC in the metaverse space.

Images