Patient treatment system and method using digital content

Abstract

The present invention provides a patient treatment system using digital content, wherein digital therapeutics are used to ensure patient compliance with treatment, thereby strengthening joint attention ability and motor imitation ability. According to the present invention, the patient's immersion in first digital content is improved through the immersion analysis based on the patient's eye tracking during output of the first digital content, thereby improving patient's joint attention ability. In addition, the patient's immersion in second digital content is improved through the immersion analysis based on estimation of the patient's posture during output of the second digital content, thereby improving the patient's motor imitation ability.

Description

Patient treatment system and method using digital content

[0001] This specification relates to digital therapeutics.

[0002] Digital therapeutics have recently been gaining attention in the digital healthcare field. Digital therapeutics, also known as digital therapeutics (DTX), refer to technologies that use software such as smartphone apps, games, VR, and chatbots as drugs to treat patients.

[0003] As chronic diseases increase due to the aging population, interest in digital therapeutics is increasing to address unmet needs caused by increased social burdens such as health insurance finances and issues with access to medical care.

[0004] These digital therapeutics, due to their software nature, have lower toxicity and side effects than existing treatments. They also don't require the manufacturing, transportation, and storage of over-the-counter drugs, making them easy to mass-produce at lower costs, potentially lowering healthcare costs. Furthermore, they allow a small number of doctors to manage a large number of patients without physical or time constraints, potentially addressing issues such as health insurance funding constraints, medical supply shortages, and regional imbalances. Furthermore, the application of digital therapeutics can partially replace in-person care for mental illness and chronic conditions, potentially reducing the risk of infection.

[0005] Thanks to these various advantages, attempts to utilize digital therapeutics to treat various conditions are increasing. For example, Republic of Korea Patent No. 2250775 discloses a digital therapeutic for the treatment of myopia.

[0006] The present invention is intended to solve the above-described problems, and its technical task is to provide a patient treatment system using digital content that can enhance joint attention and motor imitation abilities through digital therapeutics to ensure patient compliance with treatment.

[0007] According to an aspect of the present invention for achieving the above-described object, a patient treatment system using digital content comprises: a digital treatment service providing server for selecting and outputting at least one of first digital content for improving a patient's joint attention ability and second digital content for improving the patient's motor imitation ability from a digital content database; and a patient terminal for outputting at least one of the first digital content and the second digital content output from the digital treatment service providing server to the patient, wherein the patient terminal comprises: a digital content output unit for receiving at least one of the first digital content and the second digital content and outputting it to the patient; a first digital treatment unit for analyzing the patient's gaze with respect to the first digital content when the first digital content is outputted and generating a gaze movement trajectory; a second digital treatment unit for analyzing the patient's posture with respect to the second digital content when the second digital content is outputted;And an immersion determination unit that determines the similarity between the target movement trajectory of the character moving in the first digital content and the gaze movement trajectory to determine the first immersion of the patient for the first digital content, and determines the second immersion of the patient for the second digital content by determining the similarity between the posture of the patient and the posture of the character appearing in the second digital content, wherein the immersion determination unit determines that the patient is in an active immersion state (Active-Engagement) if the similarity is greater than a reference value, and determines that the patient is in a passive immersion state (Passive-Engagement) in which the patient is gazing at the screen but only listening without moving if the similarity is between a first threshold value that is less than the reference value and the reference value, and determines that the patient is in a non-immersion state (Dis-Engagement) in which the patient is away from the patient terminal or is not gazing straight ahead, and feeds back the determination result to the digital treatment service providing server.

[0008] In one embodiment, when the immersion analysis unit determines that the patient is in a non-immersive state, the digital treatment service provision server may change at least one of the size, color, and sound of a character in the first digital content or the second digital content currently being output, and when the immersion analysis unit determines that the patient is in a passive immersive state, the digital treatment service provision server may generate an alarm generation command to output to the patient an alarm including at least one of a display-type message, an audio message, and a scent to alert the patient.

[0009] Meanwhile, the first digital treatment unit generates an alarm so that the patient can approach the patient terminal when the distance between the patient terminal and the patient calculated based on the captured image of the patient is longer than the reference distance, and determines the coordinates of the focusing area where the patient's gaze is focused using a CNN (Convolution Neural Network)-based gaze detection model when the distance between the patient terminal and the patient is less than the reference distance, and calculates the gaze movement trajectory by connecting the coordinates of the focusing area determined while the first digital content is being output in time series order.

[0010] In addition, the second digital treatment unit may analyze the patient's posture using a CNN-based posture analysis model while the second digital content is output to generate skeleton data of the patient, and the immersion analysis unit may determine the second immersion by determining the similarity between the patient's skeleton data and the skeleton data of a character appearing in the second digital content.

[0011] At this time, the second digital treatment unit may extract an image of the patient from a captured image of the patient using the CNN-based posture analysis model, detect at least one of the eyes, nose, ears, shoulders, elbows, and wrists as key points in the patient image, and generate skeleton data of the patient using the key points.

[0012] In one embodiment, the digital treatment service providing server determines whether the patient is an attention deficit hyperactivity disorder (ADHD) patient, a dementia patient, or a developmentally disabled child based on at least one of the patient's biometric information, the patient's lifestyle pattern information, external environment information, internal environment information, and preliminary question-and-answer information received from the patient terminal, and selects the first digital content and the second digital content from the digital content database based on the determination result, wherein the patient's biometric information includes at least one of the patient's age, sex, body temperature, blood pressure, and brain waves, wherein the patient's lifestyle pattern information includes at least one of a sleep pattern, an exercise pattern, a meal pattern, and a work pattern, wherein the external environment information includes at least one of temperature, humidity, weather, air pressure, sunlight, and noise outside a space where the patient is located, wherein the internal environment information includes at least one of temperature, humidity, lighting, and noise inside a space where the patient is located, and wherein the preliminary question-and-answer information may be information in which the patient's answers to pre-determined questions are stored in a manner that matches each question with each other.

[0013] Meanwhile, the immersion analysis unit can calculate the patient's immersion score based on the similarity between the target movement trajectory and the gaze movement trajectory during a preset time period within the first digital content or the similarity between the patient's posture and the character's posture during a preset time period within the second digital content, and feed it back to the digital treatment service providing server.

[0014] In accordance with this embodiment, the digital treatment service providing server may provide a reward to the patient through the patient terminal when the patient's immersion score is higher than a first reference score set in advance, select new first digital content or second digital content requiring a higher immersion than the first digital content or the second digital content that was previously output and output the new first digital content or second digital content to the patient terminal, and output an alarm generation command to increase the patient's immersion to the patient terminal when the patient's immersion score is lower than a second reference score that is lower than the first reference score, and select new first digital content or second digital content requiring a lower immersion than the first digital content or the second digital content that was previously output and output the new first digital content or second digital content to the patient terminal.

[0015] In one embodiment, the immersion score may be calculated as a ratio of the total time of the time interval and the time during which the similarity between the target movement trajectory and the gaze movement trajectory or the similarity between the patient's posture and the character's posture is greater than or equal to a reference value.

[0016] According to the present invention, there is an effect that the patient's joint attention ability can be improved by improving the patient's immersion in the first digital content through immersion analysis through eye tracking when the first digital content is output.

[0017] In addition, according to the present invention, there is an effect that the patient's motor imitation ability can be improved by improving the patient's immersion in the second digital content through immersion analysis through estimation of the patient's posture when outputting the second digital content.

[0018] In addition, according to the present invention, there is an effect of ensuring treatment compliance prior to speech therapy for improving communication skills in children with developmental disabilities by improving the patient's joint attention ability and motor imitation ability.

[0019] FIG. 1 is a drawing schematically showing the configuration of a patient treatment system using digital content according to one embodiment of the present invention.

[0020] FIG. 2 is a block diagram schematically showing the configuration of a digital treatment service providing server according to one embodiment of the present invention.

[0021] Figure 3 is a block diagram schematically showing the configuration of a patient terminal according to one embodiment of the present invention.

[0022] Figure 4 is a diagram exemplarily showing a method for determining the area where a patient's gaze is focused using a gaze detection model.

[0023] FIG. 5 is a diagram exemplarily showing a method for analyzing a patient's posture by a second artificial neural network model according to one embodiment of the present invention.

[0024] Throughout the specification, identical reference numbers refer to substantially identical components. In the following description, detailed descriptions of components and functions not related to the core components of the present invention and those known in the art may be omitted. The meanings of terms used in this specification should be understood as follows.

[0025] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided solely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined solely by the scope of the claims.

[0026] The shapes, sizes, ratios, angles, numbers, etc. disclosed in the drawings for explaining embodiments of the present invention are illustrative and are not limited to the details depicted. Like reference numerals designate like elements throughout the specification. Furthermore, in describing the present invention, detailed descriptions of related known technologies will be omitted if they are deemed to unnecessarily obscure the gist of the present invention.

[0027] In this specification, when the terms "includes," "has," and "consists of," are used, other parts may be added, unless "only" is used. When a component is expressed in the singular, it includes the plural unless otherwise explicitly stated.

[0028] When interpreting a component, it is interpreted as including the error range even if there is no separate explicit description.

[0029] When describing a temporal relationship, for example, when the temporal continuity is described as 'after', 'following', 'next to', 'before', etc., it can also include cases where it is not continuous, as long as 'right away' or 'directly' is not used.

[0030] While terms like "first" and "second" are used to describe various components, these components are not limited by these terms. These terms are used merely to distinguish one component from another. Therefore, a "first" component referred to below may also be a "second" component within the technical scope of the present invention.

[0031] The individual features of the various embodiments of the present invention can be partially or wholly combined or combined with each other, and various technical linkages and operations are possible, and each embodiment can be implemented independently of each other or implemented together in a related relationship.

[0032]

[0033] Hereinafter, embodiments of the present specification will be described in detail with reference to the attached drawings.

[0034] Figure 1 is a block diagram showing the configuration of a patient treatment system using digital content according to one embodiment of the present invention.

[0035] Referring to FIG. 1, a patient treatment system (100) using digital content according to one embodiment of the present invention includes a digital treatment service providing server (110), a patient terminal (120), and a therapist terminal (130). In addition, the patient treatment system (100) using digital content according to one embodiment of the present invention may further include a guardian terminal (140). The digital treatment service providing server (110), the patient terminal (120), the therapist terminal (130), and the guardian terminal (140) may be interconnected via a wired or wireless network.

[0036] First, the digital treatment service provision server (110) provides digital content for patient treatment to the patient terminal (120), thereby providing digital content for digitally treating the patient's illness. Hereinafter, the configuration of the digital treatment service provision server (110) according to the present invention will be described in detail with reference to FIG. 2.

[0037] FIG. 2 is a block diagram schematically showing the configuration of a digital treatment service providing server according to one embodiment of the present invention. As illustrated in FIG. 2, the digital treatment service providing server (110) according to one embodiment of the present invention includes a patient status determination unit (210), a first digital content selection unit (220), a digital content DB (230), and a second digital content selection unit (240). In addition, as illustrated in FIG. 2, the digital treatment service providing server (110) according to one embodiment of the present invention may further include an event generation unit (250).

[0038] The patient status determination unit (210) receives patient information through the patient terminal (120) and determines the patient's status based on the received patient information. In one embodiment, the patient information received by the patient status determination unit (210) from the patient terminal (120) may include at least one of the patient's biometric information, the patient's lifestyle pattern information, external environment information, internal environment information, and pre-questionnaire information.

[0039] The patient's biometric information may include at least one of the patient's age, gender, body temperature, blood pressure, and brain waves, and the patient's lifestyle pattern information may include at least one of the patient's sleep pattern, exercise pattern, meal pattern, and work pattern.

[0040] The external environmental information may include at least one of temperature, humidity, weather, air pressure, sunlight, and noise outside the space where the patient is located, and the internal environmental information may include at least one of temperature, humidity, lighting, and noise inside the space where the patient is located.

[0041] Pre-questionnaire information means that the patient's answers to pre-determined questions are stored in a manner that matches the questions.

[0042] In one embodiment, the patient status determination unit (210) can determine whether the patient is a patient with Attention Deficit Hyperactivity Disorder (ADHD), a patient with dementia, or a child with a developmental disability based on patient information.

[0043] In addition, the patient status judgment unit (210) can additionally determine whether the patient is a mild attention deficit hyperactivity disorder patient or a severe mild attention deficit hyperactivity disorder patient depending on the degree of attention deficit hyperactivity disorder if the patient is a dementia patient, and can additionally determine whether the patient is a mild dementia patient or a severe dementia patient depending on the degree of dementia if the patient is a dementia patient.

[0044] The reason why the present invention determines the patient's condition through the patient condition determination unit (210) is to provide digital content optimized for the treatment of the patient based on the condition of the patient viewing the digital content.

[0045] In the above-described embodiment, the patient status determination unit (210) was described as determining the patient status based on patient information. However, in other embodiments, the patient status determination unit (210) may directly receive the patient status determination result from an external source. For example, the patient status determination unit (210) may directly receive the patient status determination result from the therapist terminal (130). In this case, the patient status determination result may include a determination result regarding whether the patient is a patient with attention deficit hyperactivity disorder, a dementia patient, or a child with a developmental disability.

[0046] The first digital content selection unit (220) selects first digital content for treating a patient's disease based on the patient's condition determined by the patient condition determination unit (210). In one embodiment, the first digital content selection unit (220) selects first digital content for improving the patient's joint attention ability based on the patient's condition from the digital content DB (230), and provides the selected first digital content to the patient terminal (120).

[0047] Joint attention refers to the ability to focus one's attention on external objects or events in conjunction with others, in order to interact or communicate with them. Joint attention can be manifested through behaviors such as detecting and following the gaze of others, exchanging gaze between others and external objects, and pointing to, giving, or showing external objects to direct others' attention to a desired object.

[0048] In the above-described embodiment, the first digital content selection unit (220) can select different first digital contents when the patient is an attention deficit hyperactivity disorder patient, a dementia patient, and a child with a developmental disability, and even when the patient is the same attention deficit hyperactivity disorder patient, different first digital contents can be selected when the patient is an adult and when the patient is a child.

[0049] The digital content database (230) stores various digital content for patient treatment. In one embodiment, the digital content may include digital content whose scenarios change based on patient interaction. For example, the digital content may increase the size or volume of a character, or change into a different character, based on the results of a patient engagement analysis if the patient's engagement is low.

[0050] The second digital content selection unit (240) selects second digital content for treating the patient's disease based on the patient's condition determined by the patient condition determination unit (210). In one embodiment, the second digital content selection unit (240) selects second digital content for improving the patient's motor imitation ability based on the patient's condition from the digital content DB (230), and provides the selected second digital content to the patient through the patient terminal (120).

[0051] Motor imitation refers to the ability to acquire and develop other people's communication skills by paying attention to them, forming mental representations of their actions, and imitating them.

[0052] In the above-described embodiment, the second digital content selection unit (240) can select different second digital contents when the patient is an attention deficit hyperactivity disorder patient, a dementia patient, or a child with a developmental disability, and even when the patient is the same attention deficit hyperactivity disorder patient, different second digital contents can be selected when the patient is an adult and when the patient is a child.

[0053] In this way, according to the present invention, the digital treatment service providing server (110) includes first and second digital content selection units (220, 240), so that digital content that can improve the patient's joint attention ability and motor imitation ability can be selected based on the patient's condition, thereby enabling digital treatment optimized for the patient to be performed.

[0054] In the above-described embodiment, the first and second digital content selection units (220, 240) are described as separate components, but in other embodiments, the first digital content selection unit (220) may perform the function of the second digital content selection unit (240) together, or the second digital content selection unit (240) may perform the function of the first digital content selection unit (220) together.

[0055] In addition, in the above-described embodiment, it was described that the first digital content and the second digital content were selected respectively to improve joint attention ability and motor imitation ability, but in another embodiment, either the first digital content selection unit (220) or the second digital content selection unit (240) may select one digital content that can improve both joint attention ability and motor imitation ability and provide it to the patient terminal (120).

[0056] Meanwhile, the digital treatment service provision server (110) may further include an event generation unit (250). When the immersion analysis result is received from the patient terminal (120), the event generation unit (250) generates an event to improve the patient's immersion according to the immersion analysis result.

[0057] In one embodiment, the event generation unit (250) may output an alarm generation command to the patient terminal (120) to increase the patient's immersion when the patient's immersion is low. For example, when the event generation unit (250) determines that the patient's immersion is low, the event generation unit (250) may provide the patient terminal (120) with an alarm generation command to generate an alarm in the form of a display message, an audio message, or a scent, and the patient terminal (120) may output an alarm according to the alarm generation command to draw the patient's attention, thereby increasing the patient's immersion.

[0058] In another embodiment, if the event generating unit (250) determines that the patient's level of immersion is low, the event generating unit (250) may cause the first or second digital content selecting unit (220, 240) to change the first or second digital content currently being provided in order to increase the patient's level of immersion. That is, the first or second digital content selecting unit (220, 240) may cause the first or second digital content selecting unit (220, 240) to output new first or second digital content that can attract the patient's attention instead of the first or second digital content that was previously being output, or may cause the first or second digital content to be provided with changed character size, color, or sound, etc., within the first or second digital content that was previously being output.

[0059] Meanwhile, if the event generation unit (250) determines that the patient's level of engagement is high, it can provide a predetermined reward to the patient.

[0060] Referring back to FIG. 1, the patient terminal (120) provides the patient with first digital content or second digital content provided from the digital treatment service provision server (110) through one or more program modules implemented in software form, thereby enabling digital treatment to be performed on the patient through the first digital content or the second digital content.

[0061] The patient terminal (120) may include a personal computer (PC) or laptop equipped with wired Internet access and browsing functions, a laptop or mobile terminal equipped with wireless LAN / mobile Internet access and browsing functions, a PCS (Personal Communication System), GSM (Global System for Mobile) terminal equipped with mobile communication network access and browsing functions, a PDA (Personal Digital Assistant), or a smart phone.

[0062] In particular, the patient terminal (120) according to the present invention analyzes the patient's level of immersion when outputting the first digital content or the second digital content, and feeds back the analysis result of the patient's level of immersion to the digital treatment service providing server (110), thereby allowing the digital treatment service providing server (110) to generate an event to increase the digital treatment effect according to the analysis result of the patient's level of immersion.

[0063] Hereinafter, a patient terminal (120) according to one embodiment of the present invention will be described in more detail with reference to FIG. 3.

[0064] FIG. 3 is a block diagram schematically showing the configuration of a patient terminal according to one embodiment of the present invention. As illustrated in FIG. 3, the patient terminal (120) may include physical components such as a processor (121) and a memory (123). In addition, the patient terminal (120) may include one or more program modules implemented in a software form configured to be executed by the processor (121), and the one or more program modules may be stored in the memory (123). The one or more program modules may include a digital content output unit (300), a first digital treatment unit (310), a first immersion analysis unit (320), a second digital treatment unit (330), and a second immersion analysis unit (340). In FIG. 3, the first immersion analysis unit (320) and the second immersion analysis unit (340) are illustrated as separate components, but this is only an example, and the first immersion analysis unit (320) and the second immersion analysis unit (340) may be implemented as a single immersion analysis unit.

[0065] The digital content output unit (300) outputs first digital content or second digital content provided from the digital treatment service provision server (110) to the patient through a display panel installed in the patient terminal (120).

[0066] The first digital treatment unit (310) analyzes the patient's gaze on the first digital content when the first digital content is output via the digital content output unit (300) to improve the patient's joint attention ability. Hereinafter, the method by which the first digital treatment unit (310) according to the present invention analyzes the patient's gaze will be described in more detail.

[0067] First, the first digital treatment unit (310) determines whether the distance between the patient terminal (120) and the patient is within the standard distance based on the captured image (CDATA) captured by the capture device (124).

[0068] The reason why the first digital treatment unit (310) determines the distance between the patient and the patient terminal (120) is that when the distance between the patient terminal (120) and the patient is within a reference distance, the area where the patient's gaze is focused can be determined using the patient's pupil information, but when the distance between the patient terminal (120) and the patient is further than the reference distance, it is difficult to accurately obtain the patient's pupil information.

[0069] Accordingly, the first digital treatment unit (310) can generate an alarm to the patient so that the patient approaches closer to the patient terminal (120) when the distance between the patient terminal (120) and the patient is longer than the reference distance.

[0070] In one embodiment, the first digital treatment unit (310) can determine the distance between the patient terminal (120) and the patient using various methods. In one embodiment, if the photographing device (124) illustrated in FIG. 1 is a device having a function capable of recognizing the distance to the patient, for example, if the photographing device (124) is implemented as a TOF camera, the first digital treatment unit (310) can determine the distance from the patient terminal (120) to the patient by comparing the phases of the light emitted from the TOF camera and the light reflected from the patient and returned.

[0071] Meanwhile, the first digital treatment unit (310) can determine a focusing area where the patient's gaze is focused using a CNN (Convolution Neural Network)-based gaze detection model when the distance between the patient terminal (120) and the patient is less than or equal to a reference distance. Specifically, the first digital treatment unit (310) can input a captured image into the gaze detection model and obtain the coordinates of the area where the patient's gaze is focused based on the first result value provided by the gaze detection model.

[0072] At this time, the gaze detection model may include a CNN-based object detection model (420) that extracts predetermined objects (415, e.g., left eye, right eye, or face) from a captured image (410), as illustrated in FIG. 4, operation models (432, 434, 436) that perform convolution operations on each of the objects extracted from the object detection model (420), and a fully connected layer (440) that connects the operation result values ​​output from the operation models (432, 434, 436) to output a first result value (450), which is a final result value. At this time, the operation models (432, 434, 436) may be implemented as a CNN.

[0073] In the above-described embodiment, it was explained that the first digital treatment unit (310) determines the area on which the patient's gaze is focused using a gaze detection model.

[0074] However, in another embodiment, the first digital treatment unit (310) may additionally detect the position of the patient's head, and thereby determine the coordinates of the area where the patient's gaze is focused by using the head position together with the first result value output from the gaze detection model. The reason why the first digital treatment unit (310) additionally uses the head position in addition to the first result value output from the gaze detection model is that the area where the patient's gaze is focused, derived based on the first result value output from the gaze detection model, may vary depending on the position of the patient's head. The head position can be detected by the second digital treatment unit (330) described below.

[0075] The first digital treatment unit (310) calculates a gaze movement trajectory of the patient's gaze by connecting the coordinates of the focusing area where the patient's gaze is focused in chronological order while the first digital content is being output. The first digital treatment unit (310) provides the calculated gaze movement trajectory to the first immersion analysis unit (320).

[0076] Meanwhile, the first digital treatment unit (310) described above can improve the accuracy of 3D Face Vertex recognition through high-quality 3D Facial Landmark configuration by learning various face (angle, position of the photographing device (124)) data sets of the patient.

[0077] The first immersion analysis unit (320) determines the first immersion of the patient for the first digital content by determining the similarity between the target movement trajectory along which the character moves within the first digital content and the gaze movement trajectory along which the patient's gaze moves. At this time, the target movement trajectory may be generated together with the first digital content when it is produced. In one embodiment, the similarity between the target movement trajectory and the gaze movement trajectory along which the patient's gaze moves may be calculated using a Euclidean distance-based similarity analysis algorithm, a Jaccard similarity analysis algorithm, or a Manhattan distance-based similarity analysis algorithm.

[0078] In one embodiment, the first immersion analysis unit (320) may determine that the patient is in an active engagement state, immersed in the first digital content, if the similarity between the target movement trajectory and the gaze movement trajectory is greater than a threshold value. In this case, the first immersion analysis unit (320) may feed back the immersion analysis results to the guardian terminal (140) and the digital treatment service provision server (110), thereby providing the patient with a predetermined reward.

[0079] On the other hand, if the similarity between the target movement trajectory and the gaze movement trajectory is between the first threshold value that is smaller than the threshold value and the threshold value, the first immersion analysis unit (320) may determine that the patient is staring at the screen but is only listening without moving, thereby determining that the patient is in a passive engagement state. In this case, the first immersion analysis unit (320) may output sound, noise, or scent to draw the patient's attention so that the patient can immerse himself in watching the first digital content. As another example, the first immersion analysis unit (320) may feed back the immersion analysis results to the guardian terminal (140) and the digital treatment service provision server (110), thereby allowing the patient to immerse himself in the first digital content again.

[0080] In addition, if the similarity between the target movement trajectory and the gaze movement trajectory is less than the first threshold, the first immersion analysis unit (320) may determine that the patient is in a disengagement state by determining that the patient has left the patient terminal (120) or is not looking straight ahead at all. In this case, the first immersion analysis unit (320) feeds back the immersion analysis results to the guardian terminal (140) and the digital treatment service provision server (110), thereby allowing the patient to immerse themselves in the first digital content again.

[0081] In the above-described embodiment, the reference value may be adjusted based on the continuous learning results of the first immersion analysis unit (320). That is, the first immersion analysis unit (320) may increase or decrease the reference value based on the test results of the general public and the test results of children with developmental disabilities for each digital content.

[0082] In particular, the first immersion analysis unit (320) feeds back the immersion analysis results to the digital treatment service provision server (110), thereby allowing the first digital content selection unit (220) to change the first digital content currently being output to new first digital content, or to change the size, color, or sound of a character within the first digital content.

[0083] In one embodiment, the first immersion analysis unit (320) may calculate a patient's immersion score based on the similarity between the target movement trajectory and the gaze movement trajectory during a preset time interval within the first digital content. In this case, the preset time interval may be included in the metadata of the first digital content and transmitted from the digital treatment service provision server (110).

[0084] For example, the first immersion analysis unit (320) can calculate the patient's immersion score as the ratio of the total time of the time section and the time during which the similarity between the target movement trajectory and the gaze movement trajectory during the time section is greater than or equal to a reference value. The first immersion analysis unit (320) feeds back the calculated patient's immersion score to the digital treatment service providing server (110), and the digital treatment service providing server (110) can provide a reward to the patient through the event generating unit (250) or vary the first digital content provided to the patient according to the patient's immersion score.

[0085] For example, the digital treatment service provision server (110) can provide a reward to the patient through the event generation unit (250) when the patient's immersion score is higher than a predetermined first reference score, and at the same time, can cause the first digital content selection unit (220) to output new first digital content that requires a higher immersion than the first digital content that was being output.

[0086] As another example, the digital treatment service provision server (110) can cause the event generation unit (250) to output an alarm generation command to increase the patient's immersion when the patient's immersion score is lower than a predetermined second reference score (a score lower than the first reference score), and at the same time, cause the first digital content selection unit (220) to output new first digital content that requires a lower immersion than the first digital content that was being output.

[0087] The second digital treatment unit (330) analyzes the patient's posture in relation to the second digital content when the second digital content is output via the digital content output unit (300) to improve the patient's motor imitation ability. Hereinafter, the method by which the second digital treatment unit (330) according to the present invention analyzes the patient's posture will be described in more detail with reference to FIG. 5.

[0088] FIG. 5 is a diagram exemplarily showing a method for analyzing a patient's posture by a second artificial neural network model according to one embodiment of the present invention.

[0089] The second digital treatment unit (330) may include, as illustrated in FIG. 5, a CNN-based object detection model (520) that extracts a patient image from a captured image (510), a cropping model (540) that performs cropping on the patient image (530) extracted from the object detection model (520), and a CNN-based keypoint detection model (560) that detects predetermined keypoints (570, e.g., both eyes, nose, ears, shoulders, elbows, wrists, etc.) from the cropped result (550). The second digital treatment unit (330) predicts the patient's posture based on the keypoints (570) detected by the keypoint detection model (560). For example, the second digital treatment unit (330) may generate skeleton data of the patient using the detected keypoints.

[0090] Although FIG. 5 illustrates that the second digital treatment unit (330) includes an object extraction model (520) and a cropping model (540) separately, object extraction and cropping may also be performed by a single model.

[0091] Meanwhile, the second digital treatment unit (330) described above can improve the accuracy of 3D posture prediction through the configuration of high-quality 3D keypoints by learning various 3D-based human posture data sets.

[0092] As described above, according to the present invention, the first and second digital treatment units (310, 330) based on CNN can be used to analyze the gaze and posture of the patient, thereby improving the joint attention ability and motor imitation ability of the patient watching the first and second digital contents, thereby enabling digital treatment of diseases of children with developmental disabilities.

[0093] The second immersion analysis unit (340) analyzes the similarity between the patient's posture analyzed by the second digital treatment unit (330) and the posture of the character appearing in the second digital content, thereby determining the patient's second immersion in the second digital content. In one embodiment, the similarity between the patient's posture and the posture of the character appearing in the second digital content can be calculated using a Euclidean distance-based similarity analysis algorithm, a Jaccard similarity analysis algorithm, or a Manhattan distance-based similarity analysis algorithm. In this case, the posture of the character appearing in the second digital content may be generated together with the production of the second digital content. For example, the posture of the character may be skeleton data of the corresponding character.

[0094] In one embodiment, the second immersion analysis unit (340) may determine that the patient is in an active engagement state, immersed in the second digital content, if the similarity between the patient's posture and the character's posture exceeds a threshold value. In this case, the second immersion analysis unit (340) may feed back the immersion analysis results to the guardian terminal (140) and the digital treatment service provision server (110), thereby providing the patient with a predetermined reward.

[0095] On the other hand, the second immersion analysis unit (340) may determine that the patient is in a passive immersion state (Passive-Engagement) if the similarity between the patient's posture and the character's posture is between the first threshold value, which is smaller than the threshold value, and the threshold value. In this case, the second immersion analysis unit (340) may output sound, noise, or scent to draw the patient's attention so that the patient can immerse himself in watching the second digital content. As another example, the second immersion analysis unit (340) may feed back the immersion analysis results to the guardian terminal (140) and the digital treatment service provision server (110), thereby allowing the patient to immerse himself in the second digital content again.

[0096] Additionally, the second immersion analysis unit (340) can determine that the patient is in a disengagement state if the similarity between the patient's posture and the character's posture is less than the first threshold. In this case, the second immersion analysis unit (340) feeds back the immersion analysis results to the guardian terminal (140) and the digital treatment service provision server (110), thereby allowing the patient to immerse themselves in the second digital content again.

[0097] In the above-described embodiment, the reference value may be adjusted based on the continuous learning results of the second immersion analysis unit (340). That is, the second immersion analysis unit (340) may increase or decrease the reference value based on the test results of the general public and the test results of children with developmental disabilities for each digital content.

[0098] In particular, the second immersion analysis unit (340) feeds back the immersion analysis results to the digital treatment service provision server (110), thereby allowing the second digital content selection unit (240) to change the second digital content currently being output to new second digital content, or to change the size, color, or sound of a character within the second digital content.

[0099] In one embodiment, the second immersion analysis unit (340) may calculate a patient immersion score based on the similarity between the patient's posture and the character's posture during a preset time interval within the second digital content. The preset time interval may be included in the metadata of the second digital content and transmitted from the digital treatment service provision server (110).

[0100] For example, the second immersion analysis unit (340) can calculate the patient's immersion score as the ratio of the total time in the time section and the time during which the similarity between the patient's posture and the character's posture is greater than a reference value. The second immersion analysis unit (340) feeds back the calculated patient's immersion score to the digital treatment service providing server (110), and the digital treatment service providing server (110) can provide a reward to the patient through the event generating unit (250) or vary the second digital content provided to the patient according to the patient's immersion score.

[0101] For example, the digital treatment service provision server (110) can provide a reward to the patient through the event generation unit (250) when the patient's immersion score is higher than a predetermined first reference score, and at the same time, can cause the second digital content selection unit (240) to output new second digital content that requires a higher immersion than the second digital content that was being output.

[0102] As another example, the digital treatment service provision server (110) can cause the event generation unit (250) to output an alarm generation command to increase the patient's immersion when the patient's immersion score is lower than a predetermined second reference score (a score lower than the first reference score), and at the same time, cause the second digital content selection unit (240) to output new second digital content that requires a lower immersion than the second digital content that was being output.

[0103] Referring back to FIG. 1, the photographing device (124) generates a photographed image (CDATA) by photographing a patient looking at the patient terminal (120). The photographing device (124) transmits the generated photographed image to the first digital treatment unit (310) and the second digital treatment unit (330). The photographing device (124) may be implemented as a camera.

[0104] In one embodiment, the imaging device (124) may be implemented as a camera capable of calculating the distance between the patient and the patient terminal (120). To calculate the distance, a camera capable of extracting the depth of the subject is required. Depth extraction techniques for a subject can be categorized into active and passive types, depending on whether light is irradiated onto the subject.

[0105] Active depth extraction methods include the Time Of Flight (TOF) method, which illuminates a subject with a laser and measures the time it takes for the light to return; the Triangulation-Based method, which utilizes the fact that the laser illuminated on the subject reaches different locations on the sensor depending on the depth; the White Light method, which illuminates a subject in the form of a line or grid and extracts depth by identifying the deformation shape; and the Structured Light method, which illuminates structured light with encoded information, identifies the structured light in the sensor, and extracts depth information using the triangulation method.

[0106] Representative passive depth extraction methods include the Depth From Stereo method, which extracts depth from parallax between images acquired from two or more camera devices, the method of extracting depth from parallax between multiple images acquired using a single camera system, and the method of extracting depth based on the change in the size of blur between two images acquired based on different optical signals.

[0107] In an embodiment of the present invention, an active, that is, a ToF (Time of flight) camera that can recognize the distance to a subject by measuring the time it takes for a light source sent to the subject to be reflected and returned is used as an example, but the present invention is not limited to this embodiment.

[0108] TOF (Time of Flight) cameras detect objects by utilizing the phase delay that occurs when light modulated with a certain frequency is reflected from the object and returned. It is widely used in fields such as topographic surveying and object attitude control. If we look at the operating principle of a TOF camera, a TOF camera includes a light source that emits light with a certain central wavelength, and the light emitted from the light source is modulated with a certain frequency and irradiated onto the object to be detected. The light irradiated onto the object is then reflected and returned to the TOF camera, and the TOF camera uses a built-in sensor to detect the returned light.

[0109] Referring back to FIG. 1, the therapist terminal (130) is connected to the digital treatment service providing server (110), the patient terminal (120), and the guardian terminal (130) via wired or wireless connections. The therapist terminal (130) provides the therapist with the first and second digital contents provided to the patient terminal (120) and the captured images captured by the capturing device (124) of the patient terminal (120), thereby enabling the therapist to monitor the patient's treatment process. The therapist terminal (130) can transmit a command input from the therapist to the digital treatment service providing server (110), the patient terminal (120), or the guardian terminal (130) based on the monitoring results of the treatment process.

[0110] For example, when a command to change the first and second digital contents is received from the therapist, the therapist terminal (130) transmits the command to the digital treatment service provision server (110) so that the first and second digital contents provided to the patient terminal (120) can be changed.

[0111] Additionally, when a command to discontinue digital therapy is received from a therapist, the therapist terminal (130) transmits the command to the digital therapy service provision server (110) and the patient terminal (120), thereby enabling the digital therapy to be discontinued. In this case, the therapist terminal (130) also transmits the command to the guardian terminal (130), thereby allowing the guardian to be informed that the patient's digital therapy has been discontinued.

[0112] Meanwhile, the therapist terminal (130) can transmit the guardian's message transmitted from the guardian terminal (130) to the therapist, thereby transmitting the guardian's requests to the therapist during the patient's treatment process.

[0113] The guardian terminal (140) receives the patient's immersion analysis results from the patient terminal (120) or the digital treatment service provision server (110) and provides them to the guardian. To this end, the patient terminal (120) or the digital treatment service provision server (110) may provide the patient's immersion analysis results to the guardian terminal (140).

[0114] In addition, the guardian terminal (140) outputs commands received from the therapist terminal (130) to the guardian, and allows the guardian to input a message if necessary, so that the guardian can convey what he or she wants to the therapist.

[0115] Those skilled in the art will appreciate that the present invention described above can be implemented in other specific forms without changing the technical idea or essential features thereof.

[0116] Additionally, the methods described herein may be implemented, at least in part, using one or more computer programs or components. These components may be provided as a series of computer instructions via a computer-readable medium or a machine-readable medium including volatile and non-volatile memory. The instructions may be provided as software or firmware, and may be implemented, in whole or in part, on hardware components such as ASICs, FPGAs, DSPs, or other similar devices. The instructions may be configured to be executed by one or more processors or other hardware components, which, when executing the series of computer instructions, perform or cause to perform all or part of the methods and procedures disclosed herein.

[0117] Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included within the scope of the present invention.

Claims

1. A digital treatment service providing server that selects and outputs at least one of the first digital content for improving the patient's joint attention ability and the second digital content for improving the patient's motor imitation ability from a digital content database; and Including a patient terminal that outputs at least one of the first digital content and the second digital content output from the digital treatment service providing server to the patient, The above patient terminal is, A digital content output unit that receives at least one of the first digital content and the second digital content and outputs it to the patient; A first digital treatment unit that analyzes the patient's gaze on the first digital content when the first digital content is output and generates a gaze movement trajectory; A second digital treatment unit that analyzes the patient's posture with respect to the second digital content when the second digital content is output; and An immersion analysis unit is included to analyze the patient's first immersion in the first digital content by determining the similarity between the target movement trajectory of the character moving in the first digital content and the gaze movement trajectory, and to analyze the patient's second immersion in the second digital content by determining the similarity between the patient's posture and the posture of the character appearing in the second digital content. The immersion analysis unit determines that the patient is in an active immersion state (Active-Engagement) if the similarity is greater than or equal to a threshold, determines that the patient is in a passive immersion state (Passive-Engagement) in which the patient is staring at the screen but only listening without moving if the similarity is between a first threshold value that is less than the threshold value, and determines that the patient is in a non-immersion state (Dis-Engagement) in which the patient has left the patient terminal or is not staring straight ahead, and feeds back the determination result to the digital treatment service providing server.

2. In paragraph 1, A patient treatment system using digital content, characterized in that the digital treatment service providing server changes at least one of the size, color, and sound of a character in the first digital content or the second digital content currently being output if the immersion analysis unit determines that the patient is in a non-immersive state.

3. In paragraph 1, The digital treatment service providing server generates an alarm generation command and outputs it to the patient terminal so that an alarm including at least one of a display-type message, an audio message, and a scent to alert the patient is output to the patient when the immersion analysis unit determines that the patient is in a passive immersive state. A patient treatment system using digital content.

4. In paragraph 1, The above first digital treatment unit, If the distance between the patient terminal and the patient calculated based on the captured image of the patient is longer than the reference distance, an alarm is generated so that the patient can approach the patient terminal. If the distance between the patient terminal and the patient is less than or equal to the reference distance, the coordinates of the focusing area where the patient's gaze is focused are determined using a CNN (Convolution Neural Network)-based gaze detection model. A patient treatment system using digital content, characterized in that the coordinates of the focusing area determined while the first digital content is being output are connected in chronological order to derive the gaze movement trajectory.

5. In paragraph 1, The second digital treatment unit analyzes the patient's posture using a CNN-based posture analysis model while the second digital content is output, thereby generating skeleton data of the patient. A patient treatment system using digital content, characterized in that the immersion analysis unit determines the second immersion by determining the similarity between the patient's skeleton data and the skeleton data of a character appearing in the second digital content.

6. In paragraph 5, A patient treatment system using digital content, characterized in that the second digital treatment unit extracts a patient image from a video of the patient using the CNN-based posture analysis model, detects at least one of the eyes, nose, ears, shoulders, elbows, and wrists as key points in the patient image, and creates skeleton data of the patient using the key points.

7. In paragraph 1, The above digital treatment service providing server, Based on at least one of the patient's biometric information, the patient's lifestyle pattern information, the external environment information, the internal environment information, and the prior questionnaire information received from the patient terminal, it is determined whether the patient is a patient with Attention Deficit Hyperactivity Disorder (ADHD), a dementia patient, or a child with a developmental disability, and the first digital content and the second digital content are selected from the digital content database according to the determination result. The patient's bio-information includes at least one of the patient's age, sex, body temperature, blood pressure, and brain waves, the patient's lifestyle pattern information includes at least one of a sleep pattern, an exercise pattern, a meal pattern, and a work pattern, the external environment information includes at least one of temperature, humidity, weather, air pressure, sunlight, and noise outside the space where the patient is located, the internal environment information includes at least one of temperature, humidity, lighting, and noise inside the space where the patient is located, and the pre-questionnaire information is information in which the patient's answers to pre-determined questions are stored in a manner that matches each question. A patient treatment system using digital content.

8. In paragraph 1, The above immersion analysis unit, A patient treatment system using digital content, characterized in that the patient's immersion score is calculated based on the similarity between the target movement trajectory and the gaze movement trajectory during a preset time period within the first digital content or the similarity between the patient's posture and the character's posture during a preset time period within the second digital content, and the result is fed back to the digital treatment service providing server.

9. In paragraph 8, The above digital treatment service providing server, If the immersion score of the patient is higher than a predetermined first reference score, a reward is provided to the patient through the patient terminal, and new first digital content or second digital content requiring a higher immersion than the first digital content or second digital content that was being output is selected as the second digital content and output to the patient terminal. A patient treatment system using digital content, characterized in that when the immersion score of the patient is lower than a second reference score that is lower than the first reference score, an alarm generation command for increasing the immersion of the patient is output to the patient terminal, and new first digital content or second digital content requiring a lower immersion than the first digital content or second digital content that was being output is selected and output to the patient terminal.

10. In paragraph 8, A patient treatment system using digital content, characterized in that the above immersion score is calculated as a ratio of the total time of the time section and the time during which the similarity between the target movement trajectory and the gaze movement trajectory or the similarity between the patient's posture and the character's posture is greater than a reference value.

Images