Server for analyzing child's psychological state using physical emotion card in ar environment and method for operation thereof

Abstract

According to various embodiments, a server for analyzing a child's psychological state using a physical emotion card in an AR environment may include a communication interface and a processor. The processor may be configured to receive recognition information regarding a physical emotion card from a user device equipped with a camera via the communication interface, transmit a control signal to display a virtual character in augmented reality (AR) on the display of the user device based on the recognition information, receive multi-modal data including voice data and gaze-tracking data of the child user collected from the user device during the interaction process between the virtual character and the child user, fuse language data obtained by converting the received voice data into text with the gaze-tracking data, and input the result into a multi-modal AI analysis engine that analyzes the child user's psychological state, identify the psychological state information of the child user output from the multi-modal AI analysis engine, and generate a psychological analysis report including the psychological state information of the child user and transmit it to a guardian terminal. Various other embodiments are also possible.

Description

Technology Field

[0001] Various embodiments of the present invention relate to a server and a method of operation for analyzing the psychological state of a child by fusionally analyzing multimodal data, such as voice and gaze, of a child user in an augmented reality (AR) environment. Background Technology

[0002] As the digital mental healthcare market grows rapidly, there is an increasing need for services to detect and intervene in children's mental health issues at an early stage. Existing psychological tests for children often rely on subjective reports or questionnaires from guardians, lacking objectivity. Furthermore, even when children participate directly, there are limitations in expressing their honest feelings due to the psychological burden of the testing environment itself.

[0003] Although psychological counseling services utilizing AI chatbots have recently emerged, most focus solely on analyzing the content of conversations with children—that is, linguistic data. However, the complex psychological state of children, particularly young children whose verbal expression is underdeveloped, is difficult to grasp through language alone. Despite the fact that non-verbal expressions—such as facial expressions, eye contact, and behavior—contain more information than words, existing technologies struggle to comprehensively collect and analyze this multi-layered data.

[0004] Therefore, there is an urgent need for new technology that can induce natural responses by lowering children's psychological defense mechanisms and dramatically improve the accuracy and reliability of psychological analysis by integrating objective and unconscious data, such as eye movements, beyond the limitations of verbal expression. In addition, there is a growing need for an educational approach that supports social development by comprehensively evaluating not only children's internal emotional states but also their understanding and coping mechanisms regarding various social situations. Prior art literature

[0005] Korean Patent Publication No. 10-2816201 (May 29, 2025) Korean Patent Publication No. 10-2683392 (July 4, 2024) The problem to be solved

[0006] The present invention aims to overcome the limitations of accuracy and reliability in existing child psychological analysis technologies by providing an augmented reality (AR) environment that is perceived by children as play rather than a test, thereby inducing spontaneous and genuine responses from children, and by comprehensively analyzing multimodal data that includes not only verbal / voice data collected during the interaction process but also non-verbal data such as eye-tracking data.

[0007] In addition, another objective of the present invention is to analyze and promote the development of children's social skills, understanding of norms, and problem-solving abilities through physical cards that present various situations containing educational lessons, going beyond simple emotional expression.

[0008] The objectives of the present invention are not limited to those mentioned above, and other unmentioned objectives can be clearly understood from the description below. means of solving the problem

[0009] According to various embodiments, a server for analyzing a child's psychological state using a real emotion card in an AR environment may be configured to include a communication interface and a processor, wherein the processor receives recognition information regarding a real emotion card from a user device equipped with a camera through the communication interface, transmits a control signal to display a virtual character in augmented reality (AR) on the display of the user device based on the recognition information, receives multi-modal data including voice data and gaze-tracking data of the child user collected from the user device during the interaction process between the virtual character and the child user, fuses the language data obtained by converting the received voice data into text with the gaze-tracking data, and inputs the result into a multi-modal AI analysis engine that analyzes the child user's psychological state, identifies the psychological state information of the child user output from the multi-modal AI analysis engine, and generates a psychological analysis report including the psychological state information of the child user and transmits it to a guardian terminal.

[0010] According to various embodiments, a method of operation of a server for analyzing a child's psychological state using a real emotion card in an AR environment may include: receiving recognition information regarding a real emotion card from a user device equipped with a camera through a communication interface of the server; transmitting a control signal to display a virtual character as augmented reality (AR) on the display of the user device based on the recognition information; receiving multi-modal data including voice data and gaze-tracking data of the child user collected from the user device during the interaction process between the virtual character and the child user; fusing the language data obtained by converting the received voice data into text with the gaze-tracking data and inputting it into a multi-modal AI analysis engine that analyzes the child user's psychological state, and identifying the child user's psychological state information output from the multi-modal AI analysis engine; and generating a psychological analysis report including the child user's psychological state information and transmitting it to a guardian terminal. Effects of the invention

[0011] The present invention can reduce children's psychological burden and defense mechanisms through play-based interaction using an augmented reality (AR) environment and real emotion cards, thereby providing the effect of securing more sincere and reliable source data.

[0012] The present invention can provide the effect of dramatically improving the accuracy and depth of psychological analysis by capturing even unconscious and non-verbal reactions, such as a child's concentration, eye avoidance, and emotional arousal, through the fusion analysis of eye tracking data obtained through a camera or sensor of a user device along with language / voice data.

[0013] The present invention can provide the effect of contributing to the healthy growth of children by enabling timely intervention by guardians through the early detection of mental health risk signals in children, such as depression and ADHD, through objective and multi-layered data analysis, and the prediction of future risk levels through the accumulation of long-term data.

[0014] The effects of the present invention are not limited to those described above, and potential effects expected from the technical features of the present invention can be clearly understood from the description below. Brief explanation of the drawing

[0015] Some of the embodiments are illustrated in the accompanying drawings so that the features of the present invention mentioned above may be understood in detail and with greater specificity through the following embodiments. Additionally, reference numerals similar to those in the drawings are intended to refer to the same or similar functions across various aspects. However, it should be noted that the accompanying drawings merely illustrate specific typical embodiments of the present invention and are not intended to limit the scope of the invention, and that other embodiments having the same effect can be sufficiently recognized. FIG. 1 illustrates a block diagram of a network environment implemented according to various embodiments of the present invention. FIG. 2 illustrates an overview of the overall operation of a child psychological test service using real emotion cards according to one embodiment of the present invention. FIG. 3 is a flowchart illustrating the operation of a server according to an embodiment of the present invention initiating interaction with a virtual character in response to a child user's selection of an emotion card, analyzing the interaction data, and providing a psychological analysis report to a guardian. FIG. 4 is a diagram illustrating examples of different types of empathetic conversations performed by a child user with a virtual character using emotion cards according to an embodiment of the present invention. FIG. 5 is a flowchart illustrating the operation of a risk prediction model in which a server according to an embodiment of the present invention predicts the potential psychological risk of a child user in the future using time series data. Specific details for implementing the invention

[0016] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail.

[0017] In describing the present invention, if it is determined that a detailed description of related prior art could unnecessarily obscure the essence of the present invention, such detailed description is omitted.

[0018] Hereinafter, various embodiments of this document are described with reference to the accompanying drawings. The embodiments and the terms used therein are not intended to limit the technology described in this document to specific embodiments and should be understood to include various modifications, equivalents, and / or substitutions of said embodiments. In relation to the description of the drawings, similar reference numerals may be used for similar components. A singular expression may include a plural expression unless the context clearly indicates otherwise. In this document, expressions such as "A or B" or "at least one of A and / or B" may include all possible combinations of items listed together. Expressions such as "first," "second," "first," or "second" may modify said components regardless of order or importance and are used only to distinguish one component from another and do not limit said components. When it is mentioned that a certain (e.g., 1st) component is "(functionally or telecommunicationally) connected" or "connected" to another (e.g., 2nd) component, said certain component may be directly connected to said other component or connected through another component (e.g., 3rd component).

[0019] In this document, "configured to" may be used interchangeably with, depending on the context, for example, hardware- or software-wise, "suitable for," "capable of," "modified to," "made to," "capable of," or "designed to." In some cases, the expression "device configured to" may mean that the device is "capable of" in conjunction with other devices or components. For example, the phrase "processor configured to perform A, B, and C" may mean a dedicated processor for performing the corresponding operations (e.g., an embedded processor), or a general-purpose processor capable of performing the corresponding operations by executing one or more software programs stored in a memory device (e.g., a CPU or application processor).

[0020] An electronic device according to various embodiments of this document may include, for example, at least one of a smartphone, tablet PC, mobile phone, video phone, e-book reader, desktop PC, laptop PC, netbook computer, workstation, server, smart mirror, PDA, PMP (portable multimedia player), MP3 player, medical device, camera, or wearable device. A wearable device may include at least one of an accessory type (e.g., watch, ring, bracelet, anklet, necklace, glasses, contact lens, or head-mounted device (HMD)), a fabric or clothing integrated type (e.g., electronic clothing), a body-attached type (e.g., skin pad or tattoo), or a bio-implantable circuit. In some embodiments, the electronic device may include, for example, a television, DVD (digital video disk) player, audio, refrigerator, air conditioner, vacuum cleaner, oven, microwave, washing machine, air purifier, set-top box, home automation control panel, security control panel, media box (e.g., Samsung HomeSync). TM , Apple TV TM , or Google TV TM ), game console (e.g., Xbox) TM PlayStation TM It may include at least one of an electronic dictionary, an electronic key, a camcorder, or an electronic photo frame.

[0021] FIG. 1 illustrates a block diagram of an electronic device and network included in a system for providing psychological analysis results according to various embodiments of the present invention. An electronic device (101) may include a bus (110), a processor (120), a memory (130), an input / output interface (150), a display (160), and a communication interface (170). In some embodiments, the electronic device (101) may omit at least one of the components or additionally include other components. The bus (110) may include a circuit that connects the components (110-170) to each other and transmits communication (e.g., control messages or data) between the components. The processor (120) may include one or more of a central processing unit, an application processor, or a communication processor (CP). The processor (120) may, for example, perform operations or data processing regarding the control and / or communication of at least one other component of the electronic device (101).

[0022] Memory (130) may include volatile and / or non-volatile memory. Memory (130) may store instructions or data related to at least one other component of the electronic device (101), for example. According to one embodiment, memory (130) may store software and / or programs (140). Programs (140) may include, for example, a kernel (141), middleware (143), an application programming interface (API) (145), and / or application programs (or "applications") (147), etc. At least part of the kernel (141), middleware (143), or API (145) may be referred to as an operating system. The kernel (141) can control or manage system resources (e.g., bus (110), processor (120), or memory (130), etc.) used to execute operations or functions implemented in other programs (e.g., middleware (143), API (145), or application program (147)). Additionally, the kernel (141) can provide an interface that controls or manages system resources by accessing individual components of the electronic device (101) from the middleware (143), API (145), or application program (147).

[0023] Middleware (143) may act as an intermediary to enable, for example, an API (145) or an application program (147) to communicate with the kernel (141) to exchange data. Additionally, middleware (143) may process one or more work requests received from the application program (147) according to priority. For example, middleware (143) may process the one or more work requests by assigning a priority to at least one of the application programs (147) to use the system resources of the electronic device (101) (e.g., bus (110), processor (120), or memory (130), etc.). The API (145) is an interface for the application (147) to control functions provided by the kernel (141) or middleware (143), and may include at least one interface or function (e.g., command) for, for example, file control, window control, image processing, or character control. The input / output interface (150) can, for example, transmit commands or data input from a user or other external device to other component(s) of the electronic device (101), or output commands or data received from other component(s) of the electronic device (101) to the user or other external device.

[0024] The display (160) may include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, a micro-electromechanical system (MEMS) display, or an electronic paper display. The display (160) may display various content (e.g., text, images, videos, icons, and / or symbols, etc.) to a user, for example. The display (160) may include a touch screen and may receive touch, gesture, proximity, or hovering input using, for example, an electronic pen or a part of the user's body. The communication interface (170) may establish communication between, for example, the electronic device (101) and an external device (e.g., a first external electronic device (102), a second external electronic device (104), or a server (106)). For example, the communication interface (170) can be connected to a network (162) via wireless or wired communication to communicate with an external device (e.g., a second external electronic device (104) or a server (106)).

[0025] Wireless communication may include cellular communication using at least one of, for example, LTE, LTE-A (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications system), WiBro (Wireless Broadband), or GSM (Global System for Mobile Communications). According to one embodiment, wireless communication may include at least one of, for example, WiFi (wireless fidelity), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, NFC (near field communication), Magnetic Secure Transmission, Radio Frequency (RF), or Body Area Network (BAN). According to one embodiment, wireless communication may include GNSS. GNSS may be, for example, GPS (Global Positioning System), Glonass (Global Navigation Satellite System), Beidou Navigation Satellite System (hereinafter "Beidou"), or Galileo, the European global satellite-based navigation system. Hereinafter, in this document, "GPS" may be used interchangeably with "GNSS". Wired communication may include at least one of, for example, USB (universal serial bus), HDMI (high definition multimedia interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).The network (162) may include at least one of a telecommunications network, for example, a computer network (e.g., LAN or WAN), the Internet, or a telephone network.

[0026] Each of the first and second external electronic devices (102, 104) may be the same or a different type of device as the electronic device (101). According to various embodiments, all or part of the operations performed on the electronic device (101) may be performed on one or more other electronic devices (e.g., electronic devices (102, 104), or a server (106). According to one embodiment, when the electronic device (101) needs to perform a function or service automatically or upon request, the electronic device (101) may request at least some of the associated functions from another device (e.g., electronic devices (102, 104), or a server (106)) instead of performing the function or service itself or additionally. The other electronic device (e.g., electronic devices (102, 104), or a server (106)) may perform the requested function or additional functions and transmit the result to the electronic device (101). The electronic device (101) may provide the requested function or service by processing the received result as is or additionally. For this purpose, for example, cloud computing, distributed computing, or client-server computing technologies may be used.

[0028] FIG. 2 illustrates an overview of the overall operation of a child psychological test service using real emotion cards according to an embodiment of the present invention.

[0029] Referring to FIG. 2, the services of the present invention may be broadly composed of educational products, services for children, and services for guardians.

[0030] First, the child user can select (①) the card that they think is most similar to their current emotional state among the multiple emotion cards included in the educational product. Afterwards, the child can use the camera of the user device (101) to recognize the selected emotion card using AR (②). If the user device (101) succeeds in recognizing the emotion card, it requests the server (106) to call the test question DB (③) based on the recognized card information, and can receive a virtual character and initial conversation data associated with the corresponding emotion from the server (106).

[0031] Through this, a virtual character is displayed in augmented reality on the display of the user device (101), and the child user can start a voice conversation (④) with the virtual character. The virtual character can ask the child questions based on the test questions received from the server (106) and receive the child's answers. The server (106) can analyze this conversation process in real time to derive a result (⑤) regarding the child's psychological state. For example, it can determine whether the child is in a risk group for depression or ADHD.

[0032] The analysis results can be generated in the form of a report (⑥) and transmitted to the guardian terminal (102). Through this report, the guardian can check the child's psychological state and refer to the results of the analysis of the causes of risk factors (⑦) analyzed by the AI. Additionally, the guardian can write questions (⑦) about the contents of the report or things they have been curious about regarding the child and send them to the server (106), which can be added as conversation topics (⑧) in future conversations with the virtual character. For example, if the guardian writes a question such as "Please ask what you think about your younger sibling being born recently," the virtual character can naturally ask a question in the next conversation session, such as "Somi, how do you feel about having a younger sibling?"

[0033] The virtual character can transmit a question written by the guardian (parent question (⑨)) to the child, receive the child's answer (answer (⑩)), and send it to the server (106). Through this interaction, the server (106) can identify the child's hidden inner thoughts, analyze them, and continuously update the psychological analysis report.

[0035] FIG. 3 is a flowchart illustrating the operation of a server (e.g., server (106) of FIG. 1) according to one embodiment of the present invention initiating interaction with a virtual character in response to a child user's selection of an emotion card, analyzing interaction data, and providing a psychological analysis report to a guardian.

[0036] According to various embodiments, the server (106) may communicate through a network (162, 164) with a user device (101) used by a child user, a guardian terminal (102) used by a guardian, and, in some cases, a professional terminal (104) used by a psychological counseling expert. The server (106) may provide an augmented reality (AR)-based psychological analysis service according to the present invention through a dedicated application or web page installed on the devices, process requests received from each device, and transmit information such as analysis results to the respective devices. According to one embodiment, the server (106) may include components for performing functions identical or similar to the components of the electronic device (101) shown in FIG. 1. Specifically, the server (106) may include one or more processors, memory, and communication modules, wherein instructions for performing the psychological analysis method of the present invention are stored in the memory, and the processor may execute the instructions to control various operations according to the present invention, such as transmitting and receiving data to and from the user device (101), guardian terminal (102), etc., and operating a multimodal AI analysis engine. According to one embodiment, the components referred to in the present disclosure as a user device (101), a guardian terminal (102), an expert terminal (104), etc., may be interpreted to extend to mean not only the physical device itself, but also each unique user account that accesses the server (106) and uses the service through the device.

[0038] In operation 301, according to various embodiments, the server (106) can receive recognition information for a real appraisal card from a user device equipped with a camera through a communication interface.

[0039] According to one embodiment, the user device (101) is equipped with a camera and may be a smartphone or a tablet PC. In this case, the user device (101) may include a front camera (first camera) with a first shooting direction generally facing the user and a rear camera (second camera) with a second shooting direction facing the external environment opposite the user. The rear camera may be used to photograph and recognize real emotion cards placed in the surrounding environment of the child user, and the front camera may be used to obtain eye tracking data by photographing the child user's face image and gaze during the interaction process.

[0040] According to one embodiment, the user device (101) may be AR (augmented reality) smart glasses. The AR smart glasses may include an external camera (second camera) that captures the external environment, and a built-in eye tracking sensor or internal camera (first camera) for tracking the movement of the user's pupils. In this case, the external camera may recognize real emotion cards, and the internal camera may perform the role of collecting gaze data of the child user more precisely.

[0041] According to one embodiment, the physical emotion card is a physical medium selected by the child user to express their emotional state. According to one embodiment, the physical emotion card is composed of multiple cards, and each card may include a unique image, color, or graphic pattern representing a specific emotion such as 'joy', 'sadness', 'anger', 'fear', 'sensitivity', etc. According to another embodiment, the physical card may further include 'situation cards' that depict various 'situations' that can provide education or lessons to the child, going beyond simply representing emotions. For example, the situation cards may include illustrations depicting situations of social interaction, compliance with norms, or problem-solving, such as 'a friend falling down and crying', 'crossing a traffic light with hands raised', 'fighting over a toy', or 'greeting an adult politely'. Additionally, each physical emotion card may include a predefined visual feature point or AR marker (e.g., QR code) so that the server (106) can uniquely identify it.

[0042] According to one embodiment, the recognition information is not the original image data itself in which the user device (101) captured the physical appraisal card, but rather structured digital data generated by the user device (101) or the server (106) analyzing the original image. Specifically, the recognition information may include at least one of the following.

[0043] 1) Unique Identifier (ID): A unique ID value indicating which emotion card the captured card is. For example, an identifier such as 'EMO_SAD_001' can be assigned to a 'Sadness' card.

[0044] 2) 3D spatial information:Information for displaying a virtual character at an accurate location in an augmented reality environment may include coordinate (x, y, z) information and direction (roll, pitch, yaw) information of a card recognized in real space.

[0046] In operation 303, according to various embodiments, the server (106) may transmit a control signal to display a virtual character as augmented reality (AR) on the display of a user device (101) based on recognition information.

[0047] According to one embodiment, the server (106) can load from memory the type, initial appearance, initial dialogue script, etc. of a virtual character that is pre-matched with the corresponding real emotion card using a unique identifier (ID) included in the recognition information. For example, if a ‘Joy’ card ID is received, an ‘energetic rabbit character’ can be selected, and if a ‘Sadness’ card ID is received, a ‘calm bear character’ can be selected. In particular, if the recognized card is a ‘situation card,’ the server (106) can load a specialized dialogue script to evaluate social skills, morality, or problem-solving ability related to the situation. For example, if a situation card ID of ‘a friend falling down and crying’ is received, the server (106) can select a dialogue script that includes questions related to empathy and helping behavior (e.g., “Your friend must be in a lot of pain. What would you like to do for them?”). Additionally, the server (106) can utilize the three-dimensional spatial information (coordinates and orientation) included in the recognition information to generate rendering information so that the virtual character appears naturally at an accurate location in real space, that is, on or next to the real emotion card, and transmit it to the user device (101).

[0048] The virtual character is a digital agent designed to perform psychological interaction with a child user and can have various forms that can give the child a sense of familiarity, such as animals, robots, people, or imaginary creatures. The virtual character is composed of 3D modeling data, textures, animation clips, sets of voice files, etc., and its behavior and speech can be controlled by a server (106).

[0049] According to one embodiment, the server (106) may be configured to display a virtual character only when it recognizes the face of a pre-registered child user through the front camera or rear camera of the user device (101). For example, when a person or object other than the registered child is seen in the camera, the character does not appear, but the moment the face of the registered child is recognized, the character appears saying "Hello, XX!", which can contribute to privacy protection and the provision of user-specific services.

[0050] According to one embodiment, when a real emotion card is recognized, the server (106) may load a virtual character by taking into account the child user's personal settings or past interaction history. For example, if the child user has previously set 'cat' as their 'favorite animal,' the default character may appear as a 'cat' regardless of which emotion card is selected. Additionally, if the child has had many positive interactions with a 'rabbit' character in the past, the 'rabbit' character may appear first to facilitate the formation of rapport with the child.

[0051] According to one embodiment, the server (106) may additionally receive surrounding environment information of the user device (101) to dynamically change the appearance of the virtual character. For example, if it is determined through the microphone of the user device (101) that the surroundings are quiet, the character may appear whispering in a small voice, saying "Quiet, quiet~". Conversely, if the surroundings are noisy or if a lot of movement is detected through the gyroscope sensor of the user device (101), the character may appear in an energetic manner, saying, "Wow! Is this an exciting party?", thereby providing interaction appropriate to the situation.

[0052] According to one embodiment, the initial appearance of a virtual character is not determined solely by the type of character, but can be dynamically transformed and displayed according to the visual characteristics of the selected emotion card. For example, if a child selects a "blue" card representing the emotion of "sadness," the virtual character's basic clothing or accessories may appear changed to a "blue" tone. If a "pointed shape" card representing the emotion of "anger" is selected, the character's hair may appear slightly spiky, intuitively showing that the character is visually empathizing with the emotion selected by the child.

[0053] According to one embodiment, when multiple physical emotion cards are recognized simultaneously, the server (106) can generate and display a new type of virtual character by fusing the characteristics of the characters corresponding to each card. For example, when a 'joy' card (rabbit character) and a 'sadness' card (bear character) are recognized together, a bear doll character with rabbit ears appears and can first recognize the child's complex emotional state and start a conversation, such as, "It's a complex feeling, being both happy and sad?"

[0054] According to one embodiment, the server (106) can analyze the initial voice tone of the child user through the microphone of the user device (101) before the virtual character appears, and change and display the character's initial tone of voice or facial expression according to the result. For example, if the child's voice is analyzed to be quiet and weak before the character appears, the virtual character can start a conversation with an empathetic attitude, such as "Is something wrong? Your voice sounds weak," along with a worried expression, from the moment the virtual character appears.

[0055] According to one embodiment, the server (106) can control the appearance method and initial behavior of a virtual character based on ambient environment data (e.g., illuminance, noise level) received from a user device (101). For example, if the surroundings are recognized as a dark night through the illuminance sensor of the user device, the virtual character may appear holding a small lamp or appear wearing pajamas to provide interactions appropriate to the temporal background.

[0056] According to one embodiment, a control signal to display a virtual character as augmented reality (AR) on a user device (101) may be transmitted in different ways depending on the type and processing capability of the user device (101). If the user device (101) is a smartphone or a tablet PC, the server (106) may transmit rendering data including 3D model data, textures, animation clips, and three-dimensional spatial information of the virtual character. In this case, the processor (120) of the user device (101) may, based on the received rendering data, overlay the virtual character onto a real background image captured in real time through a rear camera to synthesize a final AR scene and display it on a display (160). For example, when a child shines an emotion card placed on the living room floor with a smartphone, the application on the smartphone may use the data received from the server (106) to render the virtual character appearing to move on the living room floor image within the smartphone screen.

[0057] If the user device (101) is AR smart glasses, the server (106) may selectively apply a rendering processing method depending on the hardware specifications and network connection status of the glasses. According to one embodiment, if the AR smart glasses are a standalone device with its own communication module (e.g., 5G, Wi-Fi) and a high-performance processor, the server (106) may directly transmit 3D model data and rendering information of a virtual character to the glasses, similar to the case of a smartphone. In this case, the AR smart glasses may process the received data internally and project the virtual character directly into the user's actual field of vision through a transparent display or projector. For example, when a child wearing the glasses looks at an emotion card, the data transmitted from the server (106) is processed by the glasses' own computation to display it in the form of a three-dimensional hologram in the user's field of vision, as if the virtual character were actually standing on the living room floor.

[0058] According to another embodiment, the AR smart glasses may be a tethered device with limited computational capabilities that operates in conjunction with an external smartphone via Bluetooth or the like. In this case, the server (106) can transmit high-capacity 3D model data and information required for complex rendering calculations to the smartphone. The smartphone processes the data received from the server (106) to generate a video stream corresponding to the final rendering result and transmits this video stream to the AR smart glasses in real time. The AR smart glasses focus on the role of a display that projects the received video stream to the user's field of vision. For example, the server (106) sends rendering data to the smartphone, and after the smartphone calculates the entire AR scene, wirelessly transmits only the resulting screen to the glasses, allowing the glasses to operate by displaying that screen in front of the eyes. This method is advantageous for lightweighting and low-power operation of the AR smart glasses.

[0060] In operation 305, according to various embodiments, the server (106) may receive multi-modal data including voice data and gaze-tracking data of the child user collected from the user device (101) during the interaction between the virtual character and the child user.

[0061] The above multimodal data is a combination of different types of data sources, and in the present invention, may include the following:

[0062] 1) Language / Speech Data: Speech content of a child user collected through the microphone of the user device (101).

[0063] 2) Eye tracking data: Eye movements of a child user collected through a camera or sensor of a user device (101).

[0064] 3) Facial expression data: Expression change information extracted from a face image of a child user captured through a camera of a user device (101).

[0065] 4) Biosignal data: Biosignals such as heart rate and skin electrical response (GSR) collected from a wearable device (e.g., smart watch) connected to a user device (101).

[0066] The server (106) can receive and synchronize these multimodal data in real time and fuse and analyze them in a subsequent 307 operation.

[0067] According to one embodiment, the server (106) can extract linguistic features (words used, sentence structure, etc.) and acoustic features (pitching of voice tone, speed of speech, stress, tremor, etc.) together from the voice data of a child user. For example, even if a child says "I'm okay," if the word itself is positive but the voice is analyzed as trembling and the tone is low, the server (106) can collect this as data associated with a state of 'anxiety' or 'sadness'.

[0068] According to one embodiment, the server (106) can analyze the gaze time, gaze movement path, pupil size change, etc., of a specific object using eye tracking data collected through the front camera of the user device (101) or the internal sensor of the AR glasses. For example, when a virtual character asks, "What was the most fun thing you did at kindergarten?", if a pattern is detected where the child's gaze repeatedly avoids the character and goes to other places nearby, the server (106) can collect this as data indicating 'discomfort with the question' or 'a tendency to avoid'.

[0069] According to one embodiment, the server (106) can analyze the correlation between a specific utterance and a gaze response by synchronizing the language data and the eye tracking data in time. For example, if a pattern appears in which the pupil size significantly expands and the gaze stares into the void at a specific time when a child says the word 'friend,' the server (106) can collect this as data indicating that the topic of 'friend' is an important keyword that triggers high emotional arousal in the child.

[0070] According to one embodiment, the server (106) can dynamically adjust the weight of data collection based on the specific type of question presented by the virtual character. For example, when asking an open-ended question that directly asks about a child's emotions (e.g., "How do you feel right now?"), a high weight is given to language / voice data, and when visually presenting a closed-ended question that asks to choose one of two options (e.g., "Do you like A or B?"), data can be collected by giving a high weight to eye tracking data that analyzes which side the gaze goes to first and / or stays on longer.

[0071] According to one embodiment, the server (106) can collect behavioral data of the child user (e.g., nodding or shaking the head, hand movements, etc.) during the interaction process through the camera image of the user device (101) and combine it with other data for analysis. For example, if the child answers "yes" with their mouth but the camera image analysis captures a subtle shaking of the head, the server (106) can collect this as data that can be interpreted as a signal of "inconsistent response" or "suppressed denial."

[0073] In operation 307, according to various embodiments, the server (106) can identify the psychological state information of the child user output from the multimodal AI analysis engine by fusing the voice data converted into text and the eye tracking data, and then inputting it into the multimodal AI analysis engine that analyzes the psychological state of the child user.

[0074] According to one embodiment, a multimodal AI analysis engine can fuse and analyze language data converted from voice data into text and eye tracking data as primary inputs. The language data is text converted through speech recognition technology and may include words used by the child, sentence length, grammatical structure, etc. The server (106) can extract positive / negative sentiment scores, key keywords, conversation topics, etc. from the language data through natural language processing (NLP) technology.

[0075] According to one embodiment, the server (106) may perform a preprocessing process including data normalization and weight application steps to effectively fuse heterogeneous multimodal data, such as language data and eye-tracking data, having different scales and distributions. This is intended to balance the influence of each data on the model learning and inference process and to prevent the analysis results from being distorted by the scale of specific data.

[0076] According to one embodiment, the server (106) can perform normalization to convert data features having different units and ranges into a consistent scale. For example, positive / negative sentiment scores (range of -1 to +1), gaze time on a virtual character (range of 0 to thousands of ms), pupil size change rate (range of -50% to +50%), etc., have different scales in themselves, so normalization is required to prevent a specific value from having an excessively large impact.

[0077] According to one embodiment, the server (106) can convert all feature values ​​into a range between 0 and 1 by applying Min-Max Normalization. This can be calculated according to the following mathematical formula 1.

[0078]

[0079] Here, X is the original data value before normalization, and X_min and X_max are the theoretical or empirical minimum and maximum values ​​of the corresponding data feature. For example, assuming that the gaze time (X) ranges from 0ms to a maximum of 5000ms, if a gaze time of 3000ms is measured, it can be converted into a normalized value of 0.6 as (3000 - 0) / (5000 - 0) = 0.6. Similarly, if a sentiment score ranging from -1 to +1 is measured as 0.5, it is converted into a value of 0.75 as (0.5 - (-1)) / (1 - (-1)) = 1.5 / 2 = 0.75. These converted values ​​all have the same scale, allowing the AI ​​analysis engine to fairly learn the relative importance of the features.

[0080] According to one embodiment, the server (106) can generate a final fused feature vector by applying weights that consider the situational context to each normalized data feature. This is based on the idea that not all data is equally important in all situations and plays a key role in increasing the accuracy of the analysis. According to one embodiment, the server (106) can apply different weights considering the reliability of the data according to the age of the child. As a specific example, for a 3-year-old child whose language expression ability is still developing, the weight of the language data (e.g., 0.2) can be set low, and the weight of the eye-tracking data (e.g., 0.8), where unconscious reactions are clearer, can be set high.

[0081] According to one embodiment, the server (106) can dynamically change the weights in real time depending on the type of question during interaction. As a specific example, for questions that induce emotional expression, such as "Can you tell me how you felt at that time?", the weight of language / voice data can be increased, and for visual tasks that require drawing, the weight of gaze data can be increased for analysis.

[0082] According to one embodiment, the server (106) can adjust weights based on the quality or reliability of the data. As a specific example, if it is determined that the reliability of the STT conversion is low due to severe ambient noise, the weight of the voice / language data can be automatically lowered and the weight of the eye tracking data, which has relatively higher reliability, can be increased to ensure the stability of the analysis results.

[0083] After undergoing these preprocessing steps, the server (106) can temporally synchronize feature vectors extracted from different modalities and concatenate them to generate a single fused feature vector, which can be used as input for a multimodal AI analysis model. Specifically, the server (106) records a timestamp on each data stream received from the user device (101), and at a specific time (t) or a specific time interval ( For t), an operation is performed to sort each feature vector and combine them into one long vector.

[0084] According to one embodiment, the server (106) can fuse feature vectors based on the time at which a specific utterance by a child occurs. For example, during a time interval in which a voice saying "I am upset" by the child is detected, the server (106) can extract the acoustic feature vector (V_acoustic), linguistic feature vector (V_linguistic), and visual feature vector (V_visual) for that interval, respectively, and sequentially combine them to generate a final fused feature vector V_fused = Concatenate(V_linguistic, V_acoustic, V_visual).

[0085] According to one embodiment, the server (106) can generate a sequence of fused feature vectors by time intervals by fusing feature vectors based on a predetermined fixed time interval (e.g., 1 second) rather than a specific utterance time.

[0086] The fused feature vector generated above is input into a multimodal AI analysis engine and can be utilized as supporting data to comprehensively determine the authenticity of emotions expressed by the child, inner anxiety, sensitivity to specific topics, etc. The multimodal AI analysis engine can perform a task of classifying or regressing the child's psychological state using the fused data.

[0087] The output data of the above multimodal AI analysis engine may include at least one of the following as psychological state information of the child user depending on the analysis task.

[0088] 1) Psychological State Classification Results: The result of classifying the child's current psychological state into one of multiple predefined categories, such as 'stable', 'anxious', 'depressed', or 'suspected attention deficit'. This can be output as probability values ​​for each category through a softmax function in the final layer of the model.

[0089] 2) Quantitative psychological indicators:Regression results predicting specific psychological indicators, such as 'depression index', 'ADHD risk', and 'social anxiety score', as continuous numerical values ​​from 0 to 100.

[0090] The above multimodal AI analysis engine may be an artificial intelligence model pre-trained using a supervised learning method.

[0091] 1) Training data: Multiple child psychological counseling datasets accurately labeled by clinical psychologists may be used for training the above model. These datasets may consist of pairs of multimodal data (input data) collected during interaction sessions and correct labels (output data), which are the actual psychological diagnostic results of the child (e.g., whether 'anxiety disorder' was diagnosed according to DSM-5 diagnostic criteria, K-CDI test scores, etc.).

[0092] 2) Learning Method: During the training process, the AI ​​analysis engine outputs predicted values ​​for the input data and can calculate the loss or error between these predicted values ​​and the actual correct labels (e.g., Cross-Entropy for classification, MSE for regression). Subsequently, training can be completed through a process of iteratively updating the model's internal weights using backpropagation algorithms (e.g., using the Adam optimizer) in a direction that minimizes this error.

[0093] The model types of the above multimodal AI analysis engine can be implemented as one of the following or a combination thereof to effectively process the complex characteristics of multimodal data.

[0094] According to one embodiment, the AI ​​analysis engine may be a Transformer-based multimodal fusion model including an attention mechanism. This model can learn internal associations between temporally synchronized language, acoustic, and gaze features through self-attention, and learn interdependencies between different modalities (e.g., a specific word and eye avoidance at that moment) through cross-modal attention. This is advantageous for identifying which feature of a modality played a decisive role in determining a psychological state. For example, even though a positive word such as "I'm okay" was used, if the attention score is highly concentrated on the features of "low voice tone" and "unstable gaze" at that moment, the model may classify it as "anxious" by judging nonverbal signals as more important than the positive word.

[0095] According to one embodiment, the AI ​​analysis engine may be a hybrid model combining a convolutional neural network (CNN) and a recurrent neural network (RNN) family model (e.g., LSTM or GRU). In this structure, the CNN is used to extract spatial features, such as facial expression images or heatmaps visualizing gaze paths, and the LSTM or GRU may be used to extract temporal features, such as speech signals or word sequences. The high-dimensional feature vectors extracted from each model are combined and passed through a fully connected layer to derive a final psychological state classification or prediction result.

[0096] According to one embodiment, the server (106) may apply explainable AI technology to the AI ​​analysis engine. This is a technology that visually or textually presents the decisive grounds on which the model has determined a specific psychological state. For example, if it is determined that there is a "high risk of ADHD," XAI techniques such as LIME (Local Interpretable Model-agnostic Explanations) or SHAP (SHapley Additive exPlanations) can be used to include in the analysis report, such as "As a result of the analysis, the fact that the 'average gaze time with the virtual character' was short and the 'response delay time to a specific stimulus' was long served as the main grounds for judgment," thereby helping guardians and experts understand and increasing the reliability of the results.

[0097] According to one embodiment, when the recognized card is a 'situation card,' the multimodal AI analysis engine can analyze not only the child's psychological state but also the level of social cognitive ability. For example, when a situation card depicting 'a child raising their hand while crossing a traffic light' is presented and the question "Why is this child raising their hand?" is asked, the child's answer ("I want to cross the road," "I want the car to stop") is analyzed along with the gaze pattern toward the 'traffic light' and 'child raising their hand' objects on the card. If the child gives the correct answer but their gaze remains on an irrelevant background, the AI ​​analysis engine can determine this as 'recognizing the rules but having low concentration on the situation.' Conversely, if the child's gaze remains focused on the traffic light and the child's hand even without giving the correct answer, it can be analyzed as 'trying to understand the situation but still lacking proficiency in verbal expression.' Through this analysis, the server (106) can comprehensively evaluate the child's understanding of social norms, empathy for the emotions of others, problem-solving strategies, and more, beyond the child's simple emotional state.

[0098] According to one embodiment, the server (106) may accumulate and store psychological state analysis results collected from multiple interaction sessions as time-series data in memory, beyond the analysis results of a single interaction session. The server (106) may further include a risk prediction model that learns the change patterns of the accumulated time-series data to predict the potential psychological risk of a child user in the future. For example, the AI ​​may learn and detect patterns such as a gradual increase in the 'anxiety index' or a rapid increase in the frequency of 'eye avoidance' regarding a specific topic like 'friend relationships' based on the analysis results of interaction data over the past three months. When the risk prediction model detects such negative change trends, even if the current psychological state is 'stable,' it includes a prediction-based warning or recommendation message, such as "Caution: A tendency toward social anxiety has recently increased. Continuous observation is required," in the psychological analysis report transmitted to the guardian terminal (102), thereby helping the guardian recognize potential problems early and respond preemptively. The specific operation of the risk prediction model will be described in detail later in the description related to FIG. 5.

[0100] In operation 309, according to various embodiments, the server (106) may generate a psychological analysis report containing the psychological state information of the child user and transmit it to a guardian terminal through a communication interface.

[0101] The above psychological analysis report may be a comprehensive report constructed by converting raw output data of an AI analysis engine into various visual elements such as text, graphs, and images. The above report may be displayed through an application on a guardian's terminal (e.g., smartphone, tablet PC) and may include at least one of the following data and analysis information.

[0102] The aforementioned psychological analysis report may include a comprehensive analysis opinion regarding the child's current psychological state determined by a multimodal AI analysis engine. This comprehensive analysis opinion may provide professional analysis results in summary and interpretation sentences that are easy for the guardian to understand.

[0103] In addition, the above psychological analysis report may include key emotional indicators represented as quantitative scores or grades for major psychological states such as depression, anxiety, and ADHD risk.

[0104] In addition, the aforementioned psychological analysis report may include data-based grounds on which the AI ​​analysis engine determined a specific psychological state. These data-based grounds may present key data characteristics that had a decisive influence on the AI's judgment in the form of text or graphs, such as, for example, "frequency of eye avoidance on a specific topic" or "rate of negative word usage."

[0105] In addition, the above psychological analysis report may include a time-series trend showing positive or negative changes in psychological state by comparing the current analysis results with data accumulated from past interaction sessions.

[0106] In addition, the above psychological analysis report may include a customized parenting guide that suggests specific tips or recommended activities to enable the caregiver to have positive interactions with the child in daily life, based on the results of the analysis of the identified psychological state.

[0107] According to one embodiment, the psychological analysis report may apply explainable AI (XAI) technology to explicitly include the decisive grounds on which the AI ​​determined a specific psychological state. For example, along with the result "ADHD risk determined to be at the 'caution' stage" in the report, specific data-based explanations such as "Basis for judgment: (1) Average response delay time to questions from the 'virtual character' is 1.5 seconds longer than the peer average. (2) Frequency of gaze deviation during conversation is high at an average of 7 times per minute" may be provided in text, thereby helping the guardian trust the AI's judgment process and objectively understand the results.

[0108] According to one embodiment, the psychological analysis report may include a heatmap visualizing the level of gaze concentration of the child user during interaction. For example, based on the screen of the user device (101) or the field of view of the AR glasses, a heatmap image may be generated and included in the report, in which areas where the child's gaze lingers for a long time are marked in red and areas where it lingers for a short time are marked in blue. Through this, the guardian can intuitively determine whether the child is focused on the face of a virtual character or if their attention is distracted by a specific object.

[0109] According to one embodiment, the psychological analysis report may include a time-series graph representing changes in the child user's core emotions over time. For example, the X-axis of the graph may be set as interaction time and the Y-axis as 'anxiety index,' and the graph may visually show a sharp rise in the anxiety index at the point when the virtual character asks about 'friendship.' Through this, the guardian can easily recognize which topics or situations are having an emotional impact on the child.

[0110] According to one embodiment, the server (106) can dynamically adjust the expression style and the difficulty level of the terms in the report by taking into account the age and language development level of the child. For example, in a report for a guardian of a young child, simple expressions such as "my child is having a hard time" and picture icons are mainly used instead of technical terms, and in a report for a guardian of an older child, information can be provided using more specific terms such as "a tendency toward social anxiety in interpersonal relationships is observed."

[0111] According to one embodiment, the server (106) may include a section in the report that suggests a customized parenting guide or recommended activities to the guardian based on the results of the psychological analysis. For example, if the child’s ‘anxiety’ index is high, the server may provide specific and actionable solutions at the bottom of the report, such as "[Recommended Activities] Play to build a sense of security with the child: reading a book while hugging tightly, practicing deep breathing together," thereby encouraging the guardian to provide practical help based on the analysis results.

[0112] In particular, the results of the analysis on the children's social cognitive ability levels, analyzed through 'situation card' interactions, can be included in the report. For example, an item such as "[Social Development] Peer Empathy: Average Level / Rule Understanding: Needs Development" can be added to the report, and specific educational guidance linked to the analysis results can be provided, such as, "To improve rule understanding, try practicing raising your hands together when crossing the crosswalk."

[0113] According to one embodiment, the server (106) may generate a report including an interactive function that allows requesting additional information or expert interpretation regarding specific items in the report. For example, if a guardian presses the 'Learn More' button next to the 'Eye Avoidance Pattern' item in the report, a pop-up explaining the 'psychological interpretation of eye avoidance' may appear, or in the case of a paid service, it may lead to a 'Ask an expert about this part' function.

[0115] FIG. 4 is a diagram illustrating examples of different types of empathetic conversations performed by a child user with a virtual character using emotion cards according to an embodiment of the present invention.

[0116] Referring to Fig. 4, child users can engage in empathetic conversations on various topics with a virtual character displayed via AR (augmented reality). These conversations serve to lower the child's psychological defense mechanisms and induce more sincere emotional expression. For example, the virtual character can ask the child questions about social relationships, such as "What do you want to say to your friend?" or "Why did you fight with your friend?" Additionally, it can build rapport by exploring ways to regulate emotions together, such as "Shall we try singing when you feel down?" or by empathizing with the child's interests, such as "I like this food too."

[0117] These empathetic conversations can be conducted in the form of conversational diaries, video diaries, and picture diaries (HTP) depending on the type of interaction. A conversational diary is a method of recording a child's thoughts and feelings primarily through voice conversation. A video diary is a method of filming the child's appearance or surrounding environment using the camera of the user device (101) while talking, which is advantageous for collecting non-verbal expressions such as the child's behavior or facial expressions. A picture diary (HTP) can be conducted in a manner similar to the 'House-Tree-Person' drawing test, where the child draws a picture on a specific topic and a virtual character asks questions about the drawing. The server (106) can infer the child's unconscious psychological state by comprehensively analyzing the composition, color, and pen pressure of the drawing, as well as the child's description of the drawing.

[0118] According to one embodiment, the server (106) can comprehensively analyze data collected from interactions for each type of diary. For example, if a child who says "I hate my friend" in a conversational diary shows a pattern of drawing the friend much smaller than themselves or coloring them black in a drawing diary, the server (106) can determine this as a "high level of negative emotion regarding friendship" and increase the accuracy of the psychological analysis.

[0120] FIG. 5 is a flowchart illustrating the operation of a risk prediction model in which a server (106) according to an embodiment of the present invention predicts the potential psychological risk of a child user in the future using time series data.

[0121] In operation 501, according to various embodiments, the server (106) can perform the operation of collecting and preprocessing time series data to be input into a risk prediction model.

[0122] According to one embodiment, the server (106) may accumulate and store in memory psychological state information (e.g., 'depression index', 'anxiety' classification probability, etc.) identified in the 307 operation for each of a plurality of interaction sessions, along with the timestamp of the session. This is a process of constructing raw time-series data to track changes in psychological state over time. For example, data may be stored in a form such as {2024-10-01, anxiety index: 25}, {2024-10-08, anxiety index: 35}, {2024-10-15, anxiety index: 50}.

[0123] According to one embodiment, the server (106) can collect not only psychological state information but also the trend of change in specific data characteristics that appeared in each session as time-series data. For example, multidimensional time-series data can be constructed by recording the 'average eye avoidance time' when the keyword 'friend' is mentioned, or the 'frequency of use' of negative sentiment words, on a session-by-session basis. This improves the accuracy of the prediction by tracking not only the 'result' of the psychological state but also changes in microscopic data that may be the 'cause'.

[0124] According to one embodiment, the server (106) can perform missing value processing and interpolation on the collected time series data. If interaction sessions occur at irregular intervals, the values ​​at the missing points are estimated and filled in using a statistical method (e.g., linear interpolation) to convert the data into a regular time interval (e.g., one week). This is a process of processing the data to fit the input format of the time series analysis model.

[0126] In operation 503, according to various embodiments, the server (106) inputs the time series data collected and preprocessed in operation 501 into a pre-trained risk prediction model to identify the prediction results for the psychological risk that may occur in the future.

[0127] The above risk prediction model may be a time series prediction model based on an RNN (Recurrent Neural Network), and specifically, an LSTM, GRU, or Transformer-based model may be used. These models are specialized in predicting future values ​​by learning sequential patterns of past data.

[0128] The input data, output data, training data, and training method of the above risk prediction model can be specified as follows.

[0129] 1) Input data: The input data for the above-mentioned risk prediction model is multidimensional time series data collected and preprocessed from 501 operations. This may not be a single-variable time series such as 'anxiety index', but rather multivariate time series data composed of multiple variables. For example, the input vector at each time point (t) may be composed of [anxiety index, depression index, time spent avoiding eye contact on the topic of 'friends', frequency of negative word usage]. By inputting such multidimensional data together, the model can learn the complex interrelationships between variables and temporal change patterns, thereby improving the accuracy of the prediction.

[0130] 2) Output data: The output data of the above-described risk prediction model may be a prediction result regarding psychological risk over a specific future period (e.g., the next 14 weeks). According to one embodiment, the output may be a probability value (e.g., a value between 0 and 1) that a specific psychological indicator (e.g., 'depression index') exceeds a preset threshold (e.g., 70 points, 'high risk group'). For example, if time series data is input in which the 'depression index' has shown a continuous upward trend over the past three months, such as {30, 45, 40, 55, 60}, the model may output a prediction result such as "Probability that the depression index will exceed 70 points within the next month: 85%". According to another embodiment, the output may be a continuous numerical value that directly predicts the value of a specific psychological indicator at a future point in time.

[0131] 3) Training data: For training the above-mentioned risk prediction model, a time-series dataset collected from a large number of child users who have been tracked over a long period may be used. This dataset may consist of multidimensional time-series data (input data) over a specific period (e.g., the past 3 months) and pairs of whether the user actually developed into a 'high-risk group' (1 or 0) or the actual measured psychological indicator value (correct label) during a specific period thereafter (e.g., the next 1 month). Through this large-scale historical data, the model learns which time-series patterns predict future psychological risks.

[0132] 4) Learning Method: The training of the aforementioned risk prediction model is based on a supervised learning method. During the training process, the model receives input data from the training dataset, outputs a predicted value for future risk, and calculates the loss between this predicted value and the actual correct label. Subsequently, training can be completed through a process of iteratively updating the model's internal weights using the backpropagation algorithm to minimize this loss.

[0133] According to one embodiment, the risk prediction model can go beyond simply predicting future figures to detect specific 'abnormal patterns' in current time-series data and identify them as risk signals. For example, if a pattern is detected in which the 'frequency of use of positive words related to social interaction,' which had previously been stable, has decreased sharply over the past two weeks, the model can identify this as an abnormal pattern called an 'early sign of social withdrawal' and generate a warning.

[0134] According to one embodiment, the risk prediction model includes an attention mechanism to identify specific past points in time or data features that had a decisive influence on deriving the prediction result. For example, while predicting an "increase in anxiety risk," the analysis of the attention score can identify that a conversation session about "friendship" from three weeks ago and a conversation session about "academics" from one week ago had the greatest influence.

[0136] In operation 505, according to various embodiments, the server (106) may utilize the prediction results identified in operation 503 to provide meaningful information to the guardian or change the operation of the system.

[0137] According to one embodiment, the server (106) may include the identified psychological risk prediction results in a psychological analysis report and transmit it to the guardian terminal (102). For example, if a risk level of 'caution' is predicted, the report may include a prediction-based warning or recommendation message such as "Caution: Recent tendencies toward social anxiety are gradually increasing. Continuous observation is required," along with a yellow warning icon at the top of the report, to help the guardian recognize potential problems early and respond preemptively.

[0138] According to one embodiment, the server (106) may dynamically change the question configuration of the next interaction session if a specific abnormal pattern is detected in the 503 operation. For example, if a 'social withdrawal' pattern is detected, the conversation script may be adjusted so that in the next conversation, the virtual character prioritizes questions exploring friendships, such as "Have you had fun playing with your friends at kindergarten lately?" or "Have you made any new friends?"

[0139] According to one embodiment, the server (106) may provide an interface that, when the identified risk level exceeds a preset 'urgent' threshold, sends an emergency alert separate from a general report to the guardian terminal (102), or automatically suggests a referral to a professional counseling center if the guardian has given prior consent. For example, if a pattern of a surge in the frequency of use of 'self-harm-related words' is detected and an 'urgent' risk is predicted, a message such as "Urgent Alert: Expert verification of your child's psychological state may be required. Would you like to view more details?" may be immediately sent to the guardian's smartphone.

[0141] According to various embodiments, a server for analyzing a child's psychological state using a real emotion card in an AR environment may be configured to include a communication interface and a processor, wherein the processor receives recognition information regarding a real emotion card from a user device equipped with a camera through the communication interface, transmits a control signal to display a virtual character in augmented reality (AR) on the display of the user device based on the recognition information, receives multi-modal data including voice data and gaze-tracking data of the child user collected from the user device during the interaction process between the virtual character and the child user, fuses the language data obtained by converting the received voice data into text with the gaze-tracking data, and inputs the result into a multi-modal AI analysis engine that analyzes the child user's psychological state, identifies the psychological state information of the child user output from the multi-modal AI analysis engine, and generates a psychological analysis report including the psychological state information of the child user and transmits it to a guardian terminal.

[0142] According to various embodiments, the recognition information may include a unique identifier (ID) representing a unique image or marker of the physical appraisal card and three-dimensional spatial information of the physical appraisal card for displaying the virtual character at a specific location in the real space.

[0143] According to various embodiments, the processor may be configured to determine the initial appearance of the virtual character based on the unique identifier included in the recognition information, and to dynamically reflect the visual characteristics of the physical emotion card into a part of the appearance of the virtual character.

[0144] According to various embodiments, the multimodal AI analysis engine may be configured to apply different weights to the language data and the eye tracking data according to situational context, and to fuse the data to which the weights have been applied to analyze the psychological state of the child user.

[0145] According to various embodiments, the processor may be configured to further execute a risk prediction model that accumulates psychological state information identified from a plurality of interaction sessions as time series data and learns the change pattern of the accumulated time series data to predict the psychological risk that may occur in the future.

[0146] According to various embodiments, a method of operation of a server for analyzing a child's psychological state using a real emotion card in an AR environment may include: receiving recognition information regarding a real emotion card from a user device equipped with a camera through a communication interface of the server; transmitting a control signal to display a virtual character as augmented reality (AR) on the display of the user device based on the recognition information; receiving multi-modal data including voice data and gaze-tracking data of the child user collected from the user device during the interaction process between the virtual character and the child user; fusing the received voice data into text and the gaze-tracking data, and then inputting the result into a multi-modal AI analysis engine that analyzes the child user's psychological state, thereby identifying the child user's psychological state information output from the multi-modal AI analysis engine; and generating a psychological analysis report including the child user's psychological state information and transmitting it to a guardian terminal.

[0148] As used in this document, the terms “module” or “part” include a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. “Module” or “part” may be a component formed integrally or a minimum unit or part thereof that performs one or more functions. “Module” or “part” may be implemented mechanically or electronically and may include, for example, an application-specific integrated circuit (ASIC) chip, field-programmable gate arrays (FPGAs), or programmable logic device known or to be developed that performs certain operations, and may be executed by a processor (120). At least part of the device (e.g., modules or functions thereof) or method (e.g., operations) according to various embodiments may be implemented as instructions stored in a computer-readable storage medium (e.g., memory (130)) in the form of a program module. When the above instruction is executed by a processor (e.g., processor (120)), the processor may perform a function corresponding to the above instruction. Computer-readable recording media may include a hard disk, a floppy disk, a magnetic medium (e.g., magnetic tape), an optical recording medium (e.g., CD-ROM, DVD, magneto-optical medium (e.g., floptical disk), built-in memory, etc. Instructions may include code generated by a compiler or code that can be executed by an interpreter. A module or program module according to various embodiments may include at least one of the aforementioned components, some of which may be omitted, or additionally include other components. Operations performed by a module, program module, or other components according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0149] Furthermore, the embodiments disclosed in this document are presented for the purpose of explaining and understanding the disclosed technical content and are not intended to limit the scope of this disclosure. Accordingly, the scope of this disclosure should be interpreted to include all modifications or various other embodiments based on the technical concept of this disclosure.

Claims

Claim 1 A server for analyzing the psychological state of a child using a physical emotion card in an AR environment comprises a communication module and a processor, wherein the processor receives recognition information regarding a physical emotion card from a user device equipped with a camera through the communication module, wherein the recognition information includes a unique identifier (ID) representing a marker included in the physical emotion card and three-dimensional spatial information of the physical emotion card, determines a virtual character pre-matched with the physical emotion card based on the unique identifier of the recognition information, and transmits a control signal to display the virtual character in augmented reality (AR) on the display of the user device at a location within real space determined based on the three-dimensional spatial information of the recognition information, and receives multi-modal data including voice data of the child user collected from the user device and gaze-tracking data including the child user's gaze time, gaze movement path, and pupil size change, during the interaction process between the virtual character and the child user, and fuses the language data obtained by converting the received voice data into text with the gaze-tracking data by applying different weights to each other, and inputs the result into a multi-modal AI analysis engine that analyzes the psychological state of the child user. Identifying the psychological state information of the child user output from the multimodal AI analysis engine, wherein (1) the lower the age of the child user, the higher the first weight applied to the eye tracking data is set than the second weight applied to the language data, and if the type of question presented by the virtual character is a question that induces emotional expression, the second weight is set higher than the first weight, and if the type of question is a question that presents a visual task, the first weight is set higher than the second weight.(2) The multimodal AI analysis engine is trained to analyze the correlation between the words of a specific utterance and the eye-tracking data by synchronizing the language data and the eye-tracking data in time through an attention mechanism, and is configured to generate a psychological analysis report including the psychological state information of the child user and transmit it to a guardian terminal. Claim 2 delete Claim 3 In claim 1, the processor is configured to extract the visual characteristics of the physical emotion card and dynamically reflect them in a part of the appearance of the virtual character, the server. Claim 4 delete Claim 5 In claim 1, the processor is configured to accumulate the psychological state information identified from a plurality of interaction sessions as time-series data, and further execute a risk prediction model that learns the change pattern of the accumulated time-series data to predict the psychological risk that may occur in the future, a server.

Images