Method, system, and computer-readable medium for providing prediagnosis information for depression using machine learning models

Abstract

The present invention relates to a method, system, and computer-readable medium for providing preliminary depression diagnosis information using a machine learning model, and the method, system, and computer-readable medium for providing preliminary depression diagnosis information using a machine learning model, which makes judgments about depression more efficient by providing medical staff, users, counselors, etc. with analysis results on depression for reply images made by the person being evaluated and supporting information for the analysis results through a special user interface.

Description

[Technical field]

[0001] The present invention relates to a method, system, and computer-readable medium for providing preliminary depression diagnosis information using a machine learning model, and relates to a method, system, and computer-readable medium for providing preliminary depression diagnosis information using a machine learning model, in order to make judgments about depression more efficient by providing medical staff, users, counselors, etc. with analysis results on depression for reply images made by the person being evaluated and supporting information therefor through a special user interface. [Background technology]

[0002] Recently, as the COVID-19 pandemic has highlighted red flags regarding the nation's mental health, there is a growing need for more proactive suicide prevention measures. Under the current system, patients who self-diagnose themselves as having an early stage mental illness visit a psychiatric hospital to receive a diagnosis, but the reality is that it is difficult for ordinary people to easily visit a hospital to receive a diagnosis and counseling.

[0003] In particular, the suicide rate has been increasing sharply among people in their 20s and 30s, students, and women in recent years. In this younger age group, there is a psychological aversion to psychiatric treatment or aversion to having a history of psychiatric treatment recorded in their personal records, making it difficult for them to visit a psychiatric hospital.

[0004] In addition, in order for counselors to treat depression, they must know the state and severity of depression, but in most cases they rely on questionnaire surveys such as the PHQ-9. Questionnaire surveys cannot accurately describe one's own condition, and have limitations in diagnosing it.

[0005] Meanwhile, Prior Patent 1 (Korean Patent Registration No. 10-2175490) discloses a technology for providing a dialogue-based interface to a user through an artificial intelligence speaker, selecting one of predetermined sentences corresponding to a questionnaire for measuring depression based on the user's response to the dialogue provided, and providing it to the user as the dialogue content, receiving all responses to the questionnaire for measuring depression, analyzing them, and measuring depression.

[0006] In addition, Prior Patent 2 (Korean Patent Publication No. 10-2020-0042373) discloses a technology configured to acquire EEG data, detect a refined composite multi-scale permutation entropy (RCMPE) index based on the EEG data, and detect EEG variability for the EEG data using the refined composite multi-scale permutation entropy index.

[0007] However, while Prior Patent 1 provides convenience to users, it has the problem that depression cannot be accurately diagnosed simply with audio information, and Prior Patent 2 has the problem that many users need equipment to receive the diagnosis, which creates discomfort and reduces convenience for users.

[0008] Furthermore, an accurate diagnosis of depression can only be made by a doctor, and prescriptions and advice for patients are ultimately what people suffering from mental illnesses such as depression want, so methods such as those in Prior Art 1 and 2, which do not involve the activities of professional doctors, are fundamentally limited.

[0009] Meanwhile, with the recent rise of remote technology, psychiatrists and counselors can now consult with and diagnose patients through image communication. However, this type of remote technology requires that the doctor and patient meet at the same time for consultation and diagnosis, which ends up taking the same amount of time from the doctor's perspective and actually reduces the convenience of medical treatment, so it is not being actively promoted for these reasons.

[0010] On the other hand, due to recent rapid social changes, the age group that suffers from depression is expanding, and the number of people who require multiple consultations and diagnoses is increasing rapidly. However, since the number of psychiatrists is limited, it is difficult to make a diagnosis through sufficient analysis even during actual medical treatment and consultations. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] Korean Patent No. 10-2175490 [Patent Document 2] Korean Patent Publication No. 10-2020-0042373 Summary of the Invention [Problem to be solved by the invention]

[0012] The present invention aims to provide a method, system, and computer-readable medium for providing preliminary depression diagnosis information using a machine learning model to make judgments about depression more efficient by providing the results of an analysis of depression of the reply image made by the person being evaluated and the supporting information to medical staff, users, counselors, etc. through a special user interface. [Means for solving the problem]

[0013] In order to solve the above-mentioned problems, one embodiment of the present invention provides a method for providing preliminary depression diagnosis information using a machine learning model executed in a computing device having one or more processors and one or more memories, the method including: a diagnosing step of deriving preliminary depression diagnosis information for one or more reply images using the machine learning model; and a providing step of providing the preliminary depression diagnosis information to a user, wherein a first display screen displayed in the providing step includes a reply image layer in which the reply image is displayed, and a script layer in which question information regarding the reply image and reply text information extracted from the reply image are displayed, and the image time point of the reply image layer changes to a time point corresponding to the position of a script portion selected by user input in the script layer.

[0014] The first display screen includes a depression graph layer in which the degree of depression diagnosed along a time axis is displayed, and the image time point of the reply image layer changes to a time point corresponding to a time axis position selected by user input in the depression graph layer.

[0015] The diagnostic step derives the depression preliminary diagnosis information using a machine learning model based on a plurality of words extracted from audio text in the reply image, a plurality of image frames corresponding to each of the plurality of words, and a plurality of pieces of audio information corresponding to each of the plurality of words.

[0016] The diagnostic step includes a first step of extracting a plurality of words, a plurality of image frames, and a plurality of pieces of audio information extracted from audio text from the reply image; a second step of deriving a plurality of first feature information, a plurality of second feature information, and a plurality of third feature information from the plurality of words, the plurality of image frames, and the plurality of pieces of audio information using respective detailed machine learning models or algorithms; and a third step of deriving derived information having a degree of depression from the plurality of first feature information, the plurality of second feature information, and the plurality of third feature information using an artificial neural network that takes into account sequence data.

[0017] The diagnostic step derives the depression preliminary diagnosis information using a machine learning model based on two or more of a plurality of words, a plurality of image frames, and audio information extracted from the audio text extracted from the reply image.

[0018] The artificial neural network considering the sequence data corresponds to a recurrent neural network or a Transformer sequence machine learning model based on an attention mechanism, and the first feature information, the second feature information, and the third feature information corresponding to each of the plurality of words are input in a merged form to the recurrent neural network or the Transformer sequence machine learning model.

[0019] The portion of the response text displayed in the script layer is displayed with one or more of highlighting, font, size, color, and underlining changed depending on one or more of the degree of depression and the type of depression in the depression preliminary diagnosis information corresponding to the portion of the response text.

[0020] When a user selects a portion of the response text in which one or more of the highlighting, font, size, color, and underlining is changed because the degree of depression in the depression preliminary diagnosis information corresponding to the portion of the response text is equal to or above a predetermined standard, the image time point of the reply image layer changes to a time point corresponding to the position of the portion of the response text.

[0021] The depression graph layer includes a first criterion display element indicating whether the depression degree exceeds a predetermined first criterion or indicates a detailed depression type.

[0022] The first display screen further includes a scroll layer that displays scroll information of the script layer, and the scroll layer displays information display elements at each point in time that are displayed according to one or more of the degree of depression and the type of depression in the depression preliminary diagnosis information, and when a user selects an information display element, the script layer moves to the position of the response text corresponding to the selected information display element.

[0023] The second display screen displayed in the providing step includes a reply image layer in which the reply image is displayed, and a summary script layer in which summary reply text information extracted from the reply image is displayed, the summary reply text information including one or more portions of reply text in which the degree of depression in the depression preliminary diagnosis information corresponding to a portion of the reply text is equal to or above a predetermined standard, and the image time point of the reply image layer changes to a time point corresponding to the position of the script portion selected by user input in the summary script layer.

[0024] The second display screen further includes one or more selection input elements corresponding to each of the summary response text information, and a portion of a reply image corresponding to the selection input element or part of the summary response text information selected by a user's input is reproduced.

[0025] In order to solve the above problems, one embodiment of the present invention is an apparatus for providing preliminary depression diagnosis information using a machine learning model implemented in a computing device having one or more processors and one or more memories, the apparatus including: a diagnosis unit that derives preliminary depression diagnosis information using the machine learning model for one or more reply images; and a providing unit that provides the preliminary depression diagnosis information to a user. A first display screen displayed by the providing step includes a reply image layer in which the reply image is displayed, and a script layer in which question information regarding the reply image and response text information extracted from the reply image are displayed, and the image time point of the reply image layer changes to a time point corresponding to the position of a script portion selected by user input in the script layer.

[0026] In order to solve the above-mentioned problems, one embodiment of the present invention provides a computer-readable medium for implementing a method for providing preliminary depression diagnosis information using a machine learning model executed in a computing device having one or more processors and one or more memories, the method including a diagnosing step of deriving preliminary depression diagnosis information for one or more reply images using a machine learning model, and a providing step of providing the preliminary depression diagnosis information to a user, wherein a first display screen displayed in the providing step includes a reply image layer in which the reply image is displayed, and a script layer in which question information regarding the reply image and response text information extracted from the reply image are displayed, and the image time point of the reply image layer is changed to a time point corresponding to the position of a script portion selected by user input in the script layer. Effect of the Invention

[0027] According to the present invention, a method, system, and computer-readable medium can be provided that uses a machine learning model to provide preliminary depression diagnosis information, thereby enabling medical staff to make more efficient judgments about depression by providing medical staff with analysis results on depression for the reply image made by the person being evaluated and the supporting information for the same through a special user interface.

[0028] Furthermore, according to the present invention, many users in their 20s and 30s can receive a diagnosis for their depression without feeling uncomfortable.

[0029] Furthermore, according to the present invention, it is possible to diagnose whether a large number of people have depression using a kind of remote medical concept, and the diagnosis can be performed in separate time slots between the patient and the doctor, rather than each doctor having to decide on a precise time with the patient or user and performing the diagnosis in real time.

[0030] In addition, according to the present invention, the medical staff can quickly check only the important parts selected by the machine learning model without having to play back the entire image.

[0031] In addition, the present invention allows the machine learning model to automatically extract parts that it determines are important, allowing the image to be reviewed with an efficient user interface.

[0032] In addition, according to the present invention, a comprehensive judgment of depression symptoms can be provided by performing a multimodal judgment on voice features, facial expressions, and the entire text.

[0033] Furthermore, according to the present invention, rather than simply providing the judgment result of the machine learning model, information regarding the basis or explanation for the judgment of the machine learning model is provided to medical staff, thereby enabling medical staff to quickly decide whether or not to accept the opinion. [Brief description of the drawings]

[0034] [Figure 1]FIG. 1 is a diagram illustrating the overall execution steps of a system for providing preliminary depression diagnosis information according to one embodiment of the present invention. [Diagram 2] 1 is a diagram illustrating an internal configuration of a server system according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram illustrating a schematic process of a first step performed by a diagnostic unit according to an embodiment of the present invention. [Figure 4] FIG. 13 is a diagram illustrating a schematic process of a second step performed by a diagnostic unit according to an embodiment of the present invention. [Diagram 5] FIG. 13 is a diagram illustrating a schematic process of a third step performed by a diagnostic unit according to an embodiment of the present invention. [Figure 6] FIG. 2 is a diagram illustrating a first display screen according to an embodiment of the present invention. [Figure 7] FIG. 2 is a schematic diagram of a reply image layer and a script layer according to an embodiment of the present invention. [Figure 8] FIG. 2 is a schematic diagram of a reply image layer and a script layer according to an embodiment of the present invention. [Figure 9] FIG. 2 is a schematic diagram illustrating a reply image layer and a depression graph layer according to one embodiment of the present invention. [Figure 10] FIG. 4 is a schematic diagram of a second display screen according to an embodiment of the present invention. [Figure 11] 1 is a diagram illustrating a computing device corresponding to a server system, a user terminal, etc. according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] Hereinafter, the embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. In the drawings, in order to clearly explain the present invention, parts that are not related to the description are omitted, and similar parts are designated by the same reference numerals throughout the specification.

[0036] Throughout the specification, when a part is said to be "connected" to another part, this includes not only when the part is "directly connected" to another part, but also when the part is "electrically connected" to another part via another element therebetween. Furthermore, when a part is said to "include" a certain component, this does not mean that the part excludes the other component, but that the part can further include the other component, unless otherwise specified to the contrary.

[0037] In addition, terms including ordinal numbers such as first, second, etc. are used to describe various components, but the components are not limited by the terms. The terms are used only to distinguish one component from another component. For example, a first component can be referred to as a second component, and similarly, a second component can be referred to as a first component, without departing from the scope of the invention. The term "and / or" includes a combination of two or more related listed items or any one of two or more related listed items.

[0038] In this specification, the term "unit" includes a unit implemented by hardware, a unit implemented by software, and a unit implemented by both. Also, one unit may be implemented by two or more pieces of hardware, and two or more units may be implemented by one piece of hardware. Meanwhile, the term "unit" is not limited to software or hardware, and the "unit" may be configured to be in an addressable storage medium or to implement one or more processors. Thus, as an example, the "unit" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The components and functions provided by the "unit" may be combined into a smaller number of components and "units" or may be further separated into additional components and "units". Furthermore, the components and "units" may be embodied to implement one or more CPUs of either a device or a secure multimedia card.

[0039] The "patient terminal" and "medical staff terminal" mentioned below can be implemented as a computer or a portable terminal that can be connected to a server or other terminals via a network. Here, the computer includes, for example, a notebook computer, desktop, or laptop equipped with a web browser, and the portable terminal includes, for example, a wireless communication device that is guaranteed to be portable, and includes all kinds of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminals, etc. In addition, the "network" may be embodied as a wired network such as a Local Area Network (LAN), a Wide Area Network (WAN), or a Value Added Network (VAN), or any type of wireless network such as a mobile radio communication network or a satellite communication network.

[0040] FIG. 1 is a diagram illustrating the overall steps performed in a system for providing preliminary depression diagnosis information according to one embodiment of the present invention.

[0041] The method of providing depression pre-diagnosis information of the present invention is provided using a machine learning model implemented on a computing device having one or more processors and one or more memories.

[0042] In step S10, the server system 1000 provides question information to the patient terminal. The question information corresponds to information including one or more of text, voice, and image, and is presented as a media in the form of text only, image only, voice only, or voice synthesized by a virtual human agent. In this specification, a "patient" should be understood as an assessee or a diagnosis providing a reply image in the broad concept of "patient", and an object to be diagnosed with depression may be referred to as a "patient" for convenience of explanation in this specification, but here, a patient is not a patient in the concept of a hospital, but an example of an assessee. For example, when the system according to the present invention is applied in an organization such as a company, any staff member to be diagnosed should be understood as being in the category of "patient" in this specification.

[0043] Step S20 corresponds to a step in which a reply image captured by the patient terminal using the camera and MIC is transferred to the server system 1000.

[0044] In the embodiment of the present invention, after steps S10 and S20 are performed, multiple reply images may be received by the server system 1000 as steps S10 and S20 are performed again for other questions.

[0045] In step S30, a diagnosis step is performed in which a machine-learned model is used on one or more returned images to derive preliminary depression diagnosis information.

[0046] The diagnosis in this specification means deriving information on mental illness such as depression based on the patient's reply image, and should be analyzed in the broadest sense including deriving the type and degree of depression of the patient from information for determining whether the patient has depression, or deriving a basis for judging the type and degree of depression of the patient. Furthermore, the depression preliminary diagnosis information should be analyzed in the broadest sense including not only the result of the diagnosis but also detailed information, comprehensive information, and processed information related to the diagnosis.

[0047] A providing step is performed in which the depression preliminary diagnosis information is provided to a user.

[0048] The user may be a psychiatric specialist, a nurse or other medical staff member, a counselor, the person being evaluated who took the interview image, or a manager or member of an organization related to the person being evaluated's work, etc.

[0049] In step S50, the depression preliminary diagnosis information received from the medical staff terminal is displayed by a special UI. In step S50, it is assumed that the user is a medical staff member, but the scope of application of the present invention is not limited thereto, and various types of users may be provided with the depression preliminary diagnosis information. However, for the sake of convenience, the following description will be given assuming that the user who receives the depression preliminary diagnosis information is a medical staff member. In addition, in the case of the person being evaluated, the patient terminal may be the same terminal device as the medical staff terminal device, or may be displayed in the same application.

[0050] In the terminal of a user such as a medical staff member, counselor, or administrator, the result of the primary evaluation by the machine learning model and the evaluation grounds can be summarized from the depression preliminary diagnosis information without viewing the entire reply image taken by the patient terminal. In step S50 described later, the screen displayed on the medical staff terminal corresponds to a form that allows the medical staff member to intuitively grasp the condition of the corresponding patient, and in the screen displayed in this manner, the medical staff member can receive the primary preliminary information and efficiently check the screen time that he / she considers important, thereby allowing the medical staff member to provide medical treatment to the corresponding patient remotely and to use the medical treatment time efficiently.

[0051] In step S60, the medical staff inputs their own diagnosis information and transfers it to the server system 1000. The diagnosis information may include not only the diagnosis information for depression, but also information on whether the patient actually needs to visit a hospital or information on a hospital diagnosis reservation. The diagnosis information for depression is used to train the machine learning model, and is therefore utilized as input data for the model training unit 1500.

[0052] In step S70, the server system 1000 transfers a part or the whole of the diagnosis information input by the medical staff to the patient terminal. In this process, a part or the whole of the preliminary diagnosis information is provided to the patient terminal in a processed form. Through this process, the patient receives information on his / her depression diagnosed by the machine learning model and / or information on the actual diagnosis by the medical staff and / or information on the appointment with the hospital.

[0053] In some embodiments of the present invention, the server system 1000 may be configured with multiple computing systems, or the patient terminal and / or the medical staff terminal may be physically connected to the server system 1000 or may be a separate terminal.

[0054] FIG. 2 is a diagram illustrating an internal configuration of a server system 1000 according to an embodiment of the present invention.

[0055] The return image collecting unit 1100 corresponds to a module that performs steps S10 and S20. Preferably, the reply image collecting unit 1100 may perform step S10 in the form of a structured question, or may proceed to a form of selecting a question using an analysis result of the reply image of the user.

[0056] The reply images collected by the reply image collection unit 1100 in this manner are stored in the DB 1600 of the server system 1000 .

[0057] The diagnosing unit 1200 derives preliminary depression diagnosis information for one or more returned images using a machine learning model.

[0058] Specifically, the machine learning models include various detailed machine learning models that evaluate the reply images provided by the subjects (patients). The detailed machine learning models correspond to detailed machine learning models that can learn and evaluate based on deep learning, or detailed machine learning models that derive feature information for the reply images using a predetermined routine or algorithm rather than learning, and evaluate the derived feature information.

[0059] In one embodiment of the present invention, the diagnosis unit 1200 basically receives a reply image made by an assessee (patient) including a plurality of consecutive image information and voice information, and derives preliminary depression diagnosis information by a machine learning model trained by a machine learning technique such as deep learning. Furthermore, the diagnosis unit 1200 can also analyze the reply image based on a predetermined rule rather than machine learning in some steps and derive a specific evaluation value. The diagnosis unit 1200 extracts image and voice information from a reply image including a plurality of consecutive images and voices, inputs the extracted image and voice information into each detailed machine learning model, and derives a result value, or combines the image and voice information and inputs it into a detailed machine learning model to derive a result value.

[0060] The information providing unit 1300 provides the depression preliminary diagnosis information to the user, which will be described in detail later.

[0061] The notification unit 1400 corresponds to the module that performs steps S60 and S70.

[0062] The model learning unit 1500 corresponds to a module for continuously learning, by using learning data, a machine learning model used in diagnosis by the diagnosis unit 1200. Here, medical staff diagnosis information (S60) is utilized as the learning data.

[0063] Meanwhile, DB1600 stores question content for question information to be presented to the patient terminal in the reply image collection unit 1100, reply images received by the reply image collection unit 1100, a trained machine learning model for diagnosis in the diagnosis unit 1200, and notification information regarding the execution of steps S60 and S70.

[0064] In another embodiment of the present invention, the server system 1000 may include two or more servers, each of which includes a part of the configuration described above, and each of which may communicate with each other to perform a method of providing preliminary depression diagnosis information using a machine learning model. For example, a function related to the reply image collection unit 1100 may be included in a specific server, and functions related to the diagnosis unit 1200 and the model learning unit 1500 may be included in yet another specific server, and a method of providing preliminary depression diagnosis information using the machine learning model of the present invention may be performed through communication between the specific server and the yet another specific server.

[0065] FIG. 3 is a diagram illustrating a first step performed by the diagnostic unit 1200 according to an embodiment of the present invention.

[0066] In a preferred embodiment of the present invention, the diagnosis step derives the depression preliminary diagnosis information using a machine learning model based on a plurality of words extracted from audio text in the reply image, a plurality of image frames corresponding to each of the plurality of words, and a plurality of pieces of audio information corresponding to each of the plurality of words.

[0067] The first step in FIG. 3 is to extract, from the reply image, a plurality of words extracted from the voice text, a plurality of image frames corresponding to each of the plurality of words, and a plurality of pieces of voice information corresponding to each of the plurality of words.

[0068] Specifically, the first step corresponds to a process of separating image information and audio raw data from the return image. Specifically, the diagnosis unit 1200 includes an image / audio separation module, which separates the return image into image information and audio raw data.

[0069] The diagnosis unit 1200 includes an STT module, which performs Speech to Text (STT) conversion on the input reply image to derive text information for the voice of the reply image provided by the user. The STT module may use various existing STT conversion methods for the STT conversion method.

[0070] In a preferred embodiment of the present invention, a voice recognized by a voice recognizer is separated into morphemes (or words), and image data and voice raw data according to a specific section that are matched to each morpheme are extracted.

[0071] For example, in the embodiment shown in Fig. 3, a total of four morphemes, i.e. "Shini", "Tai", "Koi", and "Shishita", are derived, and image frames corresponding to each morpheme are selected. In Fig. 3, an image frame at the time when the pronunciation of the corresponding morpheme starts is selected, but this is not limited thereto, and one or more image frames related to the corresponding morpheme may be selected. This method is also applicable to speech information according to a specific section.

[0072] In other words, in the present invention, instead of treating all time sequences in the same way, sequences are treated according to the morphemes recognized by the speech recognizer, thereby improving the accuracy of multimodal analysis in the presence of additional speech information and reducing the computational load.

[0073] In the embodiment shown in FIG. 3, sequences are aligned for each morpheme recognized by the speech recognizer, and image frames and speech raw data are extracted. However, in other embodiments of the present invention, sequences are aligned for image frames, time, etc., and the remaining information is extracted, and initial data can be obtained using alignment.

[0074] On the other hand, in another embodiment of the present invention, unlike the embodiment shown in Fig. 3, alignment between image frames, audio raw data, and text (morphemes) is not performed, and each of them can be input into a multimodal model (models in Figs. 4 and 5) to derive a result. In this case, each data can be used in its entirety or sampled according to a predetermined rule.

[0075] In another embodiment of the present invention, the input data can be extracted using one or more, preferably two or more, of the image frame, the audio raw data, and the text, for example, the determination can be made only on the image frame and the audio raw data without considering the information on the text.

[0076] FIG. 4 is a diagram illustrating the process of the second step performed by the diagnosing unit 1200 according to an embodiment of the present invention.

[0077] In a second step, a step of deriving a plurality of first feature information, a plurality of second feature information, and a plurality of third feature information from the plurality of words, the plurality of image frames, and the plurality of speech information using respective detailed machine learning models is performed.

[0078] The voice information includes one or more of raw data for a voice and one or more pieces of voice feature information extracted from the raw data for the voice.

[0079] Specifically, in the second step, the words, the image frames, and the audio information are preprocessed by the respective preprocessors, and then the preprocessed data is passed to a feature information extractor, e.g., an extractor that performs an FD process on the information derived through a CNN module to extract a data string, thereby deriving first feature information, second feature information, and third feature information for each input data. The feature information extractor extracts feature information using one or more of a machine-learned model and an algorithm based on a predetermined rule. Meanwhile, in some embodiments, the preprocessor may be omitted for each of the words, image frames, and audio information.

[0080] Detailed feature information is extracted from the voice text information by a step of embedding the text information as a vector. The feature information extractor for the voice information or the RNN module in FIG. 5 can also receive the embedded vector for the question. Thus, the detailed machine learning model can derive more accurate third feature information by considering not only the reply image but also the question for the reply image.

[0081] In another embodiment of the present invention, an embedding module included in a feature information extractor for audio information represents each piece of text information in a vector form using various embedding methods such as one-hot encoding, CountVectorizer, TfidVectorizer, and Word2Vec.

[0082] In a preferred embodiment, image information, audio feature information, and text information of the reply image are input into the machine learning model, respectively, to grasp the context and intent of the user's reply in the reply image, which is difficult to grasp from text alone, thereby enabling more accurate depression diagnosis information to be derived.

[0083] Preferably, the image information (image frames) and audio raw data separated by the image and audio separation module are pre-processed individually and input to the machine learning model. A pre-processing module included in the diagnosis unit 1200 pre-processes the image information and the audio raw data. In this way, the pre-processing module converts the image information and the audio information into a form suitable for the algorithm of the machine learning model, thereby improving the performance of the machine learning model.

[0084] To this end, the pre-processing module processes missing values ​​or features for image information and audio information in a Data Cleaning step, encodes the image information and audio information into numerical data using a one-hot encoding method or the like in a Handling Text and Categorical Attributes step, converts the data in a Custom Transformers step, sets the data range in a Feature Scaling step, and automates these processes in a Transformation Pipelines step. The steps performed in the pre-processing module are not limited to the above-mentioned steps and may include various pre-processing steps for machine learning models.

[0085] FIG. 5 is a diagram illustrating a process of the third step performed by the diagnosing unit 1200 according to an embodiment of the present invention.

[0086] In a third step, a process is performed in which derived information including the degree of depression is derived from the plurality of first feature information, the plurality of second feature information, and the plurality of third feature information using an artificial neural network.

[0087] The artificial neural network includes a plurality of machine-learned inference modules, and in the case where there are four morphemes as shown in FIG. 3, first feature information of an image frame for a first morpheme, second feature information for the audio feature information for the first morpheme, and third feature information for the first morpheme text are input in a concat form to a first inference module, and the first feature information, second feature information, and third feature information for the remaining second, third, and fourth morphemes are processed in a similar manner.

[0088] In the embodiment shown in FIG. 5, the respective features input to the artificial neural network correspond to an aligned state, however, in other embodiments of the present invention, the features may be input to the artificial neural network in an unaligned form.

[0089] Preferably, the artificial neural network is an artificial neural network that considers sequence data, more preferably a recursive neural network or a transformer sequence model, in which case the first feature information, the second feature information, and the third feature information corresponding to each of the plurality of words are input in a merged form to a detail module of the artificial neural network, and more preferably, the transformer sequence model includes a transformer sequence machine learning model based on an attention mechanism.

[0090] Such a recursive neural network or transformer sequence machine learning model can finally output an inference result of a plurality of data sequences, where the plurality of data sequences includes a first diagnostic information, a second diagnostic information, ..., an Nth diagnostic information, where such diagnostic information includes depression prediction information and / or depression type.

[0091] Such an inference result corresponds to a plurality of inference results. For example, for one or more whole reply images, a plurality of words extracted from a voice text for each specific section, a plurality of image frames corresponding to each of the plurality of words, and a plurality of voice feature information corresponding to each of the plurality of words, final diagnosis information as shown in FIG. 5 can be derived.

[0092] Based on each of these final diagnosis information, it is possible to grasp the depression prediction score and / or depression type for each specific section, and when this is taken as a whole, it is possible to know which positions in the entire interview image have high depression prediction scores, or what the type of depression is when a high depression prediction score is obtained.

[0093] Preferably, in one embodiment of the present invention, the diagnosis unit 1200 may obtain depression prediction information and / or depression type in each final diagnosis information to derive overall evaluation information.

[0094] Alternatively, in another embodiment of the present invention, the final diagnosis information in FIG. 5 may be output including information on the portion where the cause predicted to be depression is located (for example, the position of the image or the position of the text).

[0095] In this manner, the present invention can provide medical staff with not only the judgment result but also detailed information on the parts on which the judgment result is based, for the entire return image or for each individual return image, allowing medical staff to quickly check only the relevant parts or play back related image parts, thereby making diagnosis more efficient.

[0096] In another embodiment of the present invention, the depression preliminary diagnosis information is derived using a machine learning model based on two or more pieces of information selected from a plurality of words, a plurality of image frames, and audio information extracted from the audio text extracted from the reply image.

[0097] In another embodiment of the present invention, the depression preliminary diagnosis information is derived using a machine learning model based on a plurality of image frames and audio information extracted from the reply image. In this case, the diagnosis unit includes a diagnosis, prediction, or inference model based on the image frames and audio information, and uses an Interpretability technique (Grad-CAM, Integrated Gradient, etc.) to check which part affected the inference result (why the score became like this), and then calculates the time, estimates the position of the corresponding text, and displays it in Figures 6, 7, 8, etc.

[0098] FIG. 6 is a diagram illustrating a first display screen according to an embodiment of the present invention.

[0099] The method for providing preliminary depression diagnosis information of the present invention includes a step of providing preliminary depression diagnosis information to a user.

[0100] The first display screen displayed in the providing step includes a reply image layer (L1) in which the reply image is displayed, and a script layer (L4) in which question information regarding the reply image and reply text information extracted from the reply image are displayed, and the image time point of the reply image layer (L1) is changed to a time point corresponding to the position of a script portion selected by user input in the script layer (L4).

[0101] The first display screen can be displayed on a terminal device of the medical staff or, if necessary, on a terminal device of the person to be evaluated (patient).

[0102] The reply image layer (L1) corresponds to a layer that plays back reply images, and users (medical staff, counselors, administrators, etc.) can select play / stop, control the playback speed, and change the playback point in the image. These functions are performed by inputting the control element (E1).

[0103] The script layer (L4) displays the patient's voice text converted into text through STT in the diagnosis unit 1200. In this way, the medical staff can quickly check the response content of the patient, the person to be diagnosed, or the person to be evaluated, and when they click or select a part of the response content that they consider important, the image time point of the reply image layer (L1) changes to the corresponding image time point, so that they can quickly check the actual image.

[0104] Preferably, the first display screen includes a depression graph layer (L2) in which the degree of depression diagnosed along a time axis is displayed, and in the depression graph layer (L2), the image time point of the reply image layer (L1) is changed to a time point corresponding to a time axis position selected by user input.

[0105] In such an interface, when a user selects a part that is judged to have a high level of depression in the depression graph layer (L2), the playback point of the image in the return image layer (L1) automatically changes, and the user can selectively check only the parts that the machine learning model judges to have depressive elements, allowing the user to quickly check the important parts related to depression without having to look at the entire return image. In addition, medical staff can personally judge the reasons for the judgment derived by the machine learning model and decide whether or not to admit the patient.

[0106] Preferably, the portion of the response text displayed on the script layer (L4) is displayed with one or more of highlighting, font, size, color, and underlining changed depending on one or more of the degree of depression and the type of depression in the depression preliminary diagnosis information corresponding to the portion of the response text.

[0107] In this manner, medical staff can grasp the reason for the judgment made by the diagnosis unit 1200 and can intuitively grasp whether the degree of depression is above a predetermined standard or the text portion related to depression in the diagnosis unit 1200, and can quickly play back the image of the relevant portion that is judged to be important without playing back the entire returned image.

[0108] Desirably, the depression graph layer (L2) includes a first criterion display element (E3) indicating that the degree of depression exceeds a predetermined first criterion, and a second criterion display element (E4) indicating that the degree of depression exceeds a predetermined second criterion.

[0109] The first criterion display element (E3) corresponds to the border of mild depression, and the second criterion display element (E4) corresponds to the border of severe depression. With this display interface, the user can efficiently search for the part that exceeds the reference value for the degree of depression. That is, the first criterion display element and the second criterion display element can display the detailed type of depression.

[0110] The first display screen further includes a scroll layer (L5) that displays scroll information of the script layer (L4), and the scroll layer (L5) displays information display elements (E6) at each point in time that are displayed according to one or more of the degree of depression and the type of depression in the depression preliminary diagnosis information, and when a user selects an information display element (E6), the script layer (L4) moves to the position of the response text corresponding to the selected information display element (E6).

[0111] Preferably, the color, size, shape, etc. of the information display element (E6) varies depending on at least one of the severity and type of depression, thereby enabling medical staff to intuitively determine the overall severity of depression of the patient graphically and quickly move to a location where the severity of depression is deemed severe.

[0112] Also, if the text content of the reply image is long, the entire text cannot be displayed on one screen. In this case, the user can use the scroll layer (L5) to move to the desired text and image playback point.

[0113] The page selection layer (L3) provides an interface that allows input of changes to pages relative to the whole as in FIG. 6 and to pages relative to the summary as in FIG.

[0114] This configuration of the first display screen allows medical staff to check the important parts without having to check multiple return images individually, and also allows them to quickly check whether the basis for the judgment made by the machine learning model is valid and to grasp the overall evaluation results of the images over a long period of time, thereby enabling more accurate and efficient diagnosis of depression.

[0115] FIG. 7 is a schematic diagram of a reply image layer (L1) and a script layer (L4) according to one embodiment of the present invention, and FIG. 8 is a schematic diagram of a reply image layer (L1) and a script layer (L4) according to one embodiment of the present invention.

[0116] As shown in Fig. 7, the reply image layer (L1) displays one of the reply images at a specific time point, while the script layer (L4) displays text information of the patient's response to each question.

[0117] As described above, depending on one or more of the types and degrees of depression diagnosed by the diagnosis unit 1200, the format in which the text is displayed changes as shown in Figure 7, and this is also reflected in the scroll layer (L5) to display the information display element (E6).

[0118] The portion of the response text that is displayed with one or more of the highlight, font, size, color, and underline changed is when the degree of depression in the depression preliminary diagnosis information corresponding to the portion of the response text is equal to or above a predetermined standard, and when a user selects the portion of the response text that is displayed with one or more of the highlight, font, size, color, and underline changed, the image time point of the reply image layer (L1) changes to a time point corresponding to the position of the corresponding portion of the response text.

[0119] 7, the highlight colors at the top and bottom correspond to colors selected according to the type of depression, such as melancholia, atypical depression, adolescent depression, geriatric depression, depression associated with alcoholism, and menopausal depression.

[0120] The screen in Figure 7 shows a case where the image playback point in the reply image layer (L1) changes when the upper part of the script layer (L4) is selected and input, and the screen in Figure 8 shows a case where the image playback point also changes when the lower part of the script layer (L4) is selected and input, changing video number 1 in the reply image layer (L1) to number 2.

[0121] FIG. 9 illustrates a schematic representation of a reply image layer (L1) and a depression graph layer (L2) according to one embodiment of the present invention.

[0122] As described above, the first display screen includes a depression graph layer (L2) in which the degree of depression diagnosed along a time axis is displayed, and in the depression graph layer (L2), the image and / or image time point of the reply image layer (L1) changes to the image and / or time point corresponding to the time axis position selected by user input.

[0123] FIG. 10 illustrates a schematic diagram of a second display screen according to an embodiment of the present invention.

[0124] Preferably, the second display screen displayed in FIG. 10 corresponds to the screen displayed when the user selects the summary portion from the page selection layer (L3).

[0125] The second display screen displayed in the providing step includes a reply image layer (L1) in which the reply image is displayed, and a summary script layer (L6) in which summary response text information extracted from the reply image is displayed.

[0126] Preferably, the summary response text information includes one or more portions of the response text in which the degree of depression in the depression preliminary diagnosis information corresponding to the portion of the response text is equal to or greater than a predetermined standard.

[0127] Preferably, the image time point of the reply image layer (L1) is changed to a time point corresponding to the position of a script portion selected by a user's input in the summary script layer (L6).

[0128] The above-mentioned scroll layer (L5) is also displayed on such a second display screen.

[0129] The second display screen further includes one or more selection input elements corresponding to each of the summary response text information, and a portion of a plurality of reply images corresponding to the selection input element or a portion of the summary response text information selected by a user's input is reproduced.

[0130] Preferably, the second display screen further includes one or more check boxes (E7) corresponding to each of the summary response text information. For example, in Fig. 10, each check box (E7) is displayed at a position corresponding to each of the summary response text information. The check boxes (E7) are an example of the selection input elements.

[0131] Preferably, a plurality (or one or more) reply image portions corresponding to a plurality of pieces of summary reply text information for which the user has selected the checkboxes (E7) are played back in response to a user input. Alternatively, in another embodiment of the present invention, a plurality (or one or more) pieces of summary reply text information, i.e., sentences, etc., are selected and the corresponding reply image portions are played back.

[0132] Specifically, in one embodiment of the present invention, when a user selects an element (E8) displayed for playback after selecting a check box (E7) for a script that the user wishes to play in the summary script layer (L6), the script for which the check box (E7) is selected is played back. Such playback can also be performed in a repeated playback mode.

[0133] That is, on the second display screen, the diagnostic unit 1200 collects and displays the scripts that are judged to have a high level of depression, and in this state, only the part selected by the user is reproduced, that is, the image part that the user secondarily selected with respect to the inference result of the machine learning model, so that the medical staff can make a diagnosis more easily and make a more accurate judgment using the judgment result of the machine learning model without having to check the entire reply image. In the case of a counselor, by more clearly grasping which part has a problem, the counselor can give a more accurate consultation to the counselor.

[0134] FIG. 11 illustrates a computing device including a server system, a user terminal, and the like according to an embodiment of the present invention.

[0135] For example, the server system 1000 in FIG. 2 above includes the components of the computing device in FIG.

[0136] 11, a computing device 11000 includes at least one processor 11100, a memory 11200, a peripheral device interface 11300, an input / output subsystem 11400, a power circuit 11500, and a communication circuit 11600. Here, the computing device 11000 corresponds to the server system 1000 in FIG. 1 or one or more servers included in the server system 1000.

[0137] The memory 11200 may include, for example, a high-speed random access memory, a magnetic disk, an SRAM, a DRAM, a ROM, a flash memory, or a non-volatile memory. The memory 11200 may include software modules, a set of commands, or other various data required for the operation of the computing device 11000.

[0138] Here, access to the memory 11200 from other components such as the processor 11100 and the peripheral device interface 11300 is controlled by the processor 11100.

[0139] The peripheral device interface 11300 couples input and / or output peripheral devices of the computing device 11000 to the processor 11100 and the memory 11200. The processor 11100 executes software modules or sets of commands stored in the memory 11200 to perform various functions and process data for the computing device 11000.

[0140] The I / O subsystem couples various I / O peripherals to the peripheral interface 11300. For example, the I / O subsystem may include controllers for coupling peripherals such as a monitor, keyboard, mouse, printer, and, if desired, a touch screen or sensor light to the peripheral interface 11300. In another aspect, I / O peripherals may be coupled to the peripheral interface 11300 without going through the I / O subsystem.

[0141] The power circuit 11500 may provide power to all or some of the components of the terminal. For example, the power circuit 11500 may include a power management system, one or more power sources such as a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, or distribution.

[0142] The communications circuitry 11600 allows for communication with other computing devices using at least one external port.

[0143] Alternatively, as previously mentioned, if desired, the communications circuitry 11600 may include RF circuitry to enable communication with other computing devices by sending and receiving RF signals, also known as electromagnetic signals.

[0144] The embodiment of Fig. 11 is merely one example of the computing device 11000, and the computing device 11000 may omit some components in Fig. 11, include additional components not shown, or have a configuration or arrangement in which two or more components are combined. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen, a sensor, etc. in addition to the components in Fig. 11, and the communication circuit 11600 may include circuits for RF communication of various communication methods (WiFi, 3G, LTE, Bluetooth, NFC, Zigbee, etc.). The components included in the computing device 11000 may be embodied in hardware, software, or a combination of both hardware and software, including one or more signal processing or application-specific integrated circuits.

[0145] Although the method and system for providing preliminary diagnostic information of the present invention have been described above with reference to depression, the present invention may be used for a variety of mental disorders, in which case the term "depression" may be replaced with "mental disorder."

[0146] Methods according to embodiments of the present invention are embodied in the form of program instructions executed through various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be a PC-based program or an application dedicated to mobile terminals. The application to which the present invention is applied is installed in a user terminal or a member store terminal through a file provided by a file distribution system. As an example, the file distribution system includes a file transfer unit (not shown) that transfers the file upon request from the user terminal or the member store terminal.

[0147] The devices described above may be implemented using hardware components, software components, and / or a combination of hardware and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, the processing device may be described as being one in number, but those skilled in the art will appreciate that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0148] Software includes computer programs, codes, instructions, or any combination of one or more of these, which can configure or instruct a processing device to perform a desired operation, either individually or collectively. The software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be analyzed by the processing device or to provide instructions or data to the processing device. The software can also be distributed on computing devices coupled by a network, and stored or executed in a distributed manner. The software and data are recorded on one or more computer-readable recording media.

[0149] The method according to the embodiment may be embodied in the form of program instructions executed by various computer means and recorded on a computer readable medium. The computer readable medium may include, alone or in combination, program instructions, data files, data structures, and the like. The program instructions recorded on the medium may be specially designed and constructed for the embodiment, or may be known and available to those skilled in the art of computer software. Computer readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as protocol disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Program instructions include not only machine language code, such as produced by a compiler, but also high-level language code executed by a computer using an interpreter, and the like. The hardware devices may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0150] According to one embodiment of the present invention, a method, system, and computer-readable medium can be provided that provide preliminary depression diagnosis information using a machine learning model to provide medical staff with analysis results for depression on the reply image made by the person being evaluated and supporting information therefor through a special user interface, thereby making it possible for medical staff to make more efficient judgments about depression.

[0151] According to one embodiment of the present invention, many users in their 20s and 30s can receive a diagnosis for their depression without feeling uncomfortable.

[0152] According to one embodiment of the present invention, as a kind of remote medical concept, it is possible to diagnose whether a large number of people have depression, and the diagnosis can be performed in separate time zones between the patient and the doctor, rather than in real time at a precise time determined by each doctor with the patient or user.

[0153] According to one embodiment of the present invention, medical staff can quickly check only the important parts selected by the machine learning model without having to play back the entire image.

[0154] According to one embodiment of the present invention, a machine learning model can automatically extract portions that are deemed important and review the image in an efficient user interface.

[0155] In one embodiment of the present invention, a multi-modal judgment can be made on voice features, facial expressions, and the entire text to provide a comprehensive judgment on depression symptoms.

[0156] In one embodiment of the present invention, rather than simply providing the judgment result of the machine learning model, information regarding the basis or explanation for the machine learning model's judgment is provided to medical staff, enabling medical staff to quickly decide whether or not to accept the opinion.

[0157] An embodiment of the present invention may also be embodied in the form of a recording medium containing computer executable commands, such as program modules executed by the computer. Computer readable media is any available medium that can be accessed by a computer, including volatile and non-volatile media, separate and non-separate media. Computer readable media also includes computer storage media and communication media. Computer storage media includes volatile and non-volatile, separate and non-separate media embodied in any method or technology for storage of information, such as computer readable commands, data structures, program modules, or other data. Communication media typically includes computer readable commands, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transport mechanism, and includes any information delivery media.

[0158] Although the method and system of the present invention have been described with respect to specific embodiments, some or all of its components or operations may be implemented using a computer system having a general-purpose hardware architecture.

[0159] The above description of the present invention is merely illustrative, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not limiting. For example, each component described as a single type can be implemented in a distributed manner, and similarly, each component described as distributed can be implemented in a combined form.

[0160] The scope of the present invention is indicated by the claims set forth below rather than by the above detailed description, and all modifications and variations that fall within the meaning and scope of the claims and their equivalents should be interpreted as being included within the scope of the present invention.

Claims

1. 1. A method for providing pre-depression diagnosis information using a machine learning model implemented on a computing device having one or more processors and one or more memories, comprising: presenting query information to a patient terminal from the computing device; a reply image collecting step of collecting reply images including image information and voice information of responses of the patient to the question information; a diagnosis step of deriving preliminary depression diagnosis information from one or more of the reply images using a machine learning model; providing the depression preliminary diagnosis information to a user; The first display screen displayed in the providing step includes: a reply image layer on which the reply image is displayed; a script layer on which the question information presented to the patient terminal and reply text information extracted from the reply image are displayed, The method for providing preliminary depression diagnosis information, wherein the image time point of the reply image layer is changed to a time point corresponding to the position of a script portion selected by a user's input in the script layer.

2. The first display screen includes a depression graph layer that displays a diagnosed depression level along a time axis, The method for providing preliminary depression diagnosis information according to claim 1 , wherein the image time point of the reply image layer is changed to a time point corresponding to a time axis position selected by a user's input in the depression graph layer.

3. 2. The method for providing preliminary depression diagnosis information according to claim 1, wherein the diagnosis step derives the preliminary depression diagnosis information using a machine learning model based on a plurality of words contained in text extracted from the voice included in the reply image, a plurality of image frames corresponding to each of the plurality of words, and a plurality of pieces of voice information corresponding to each of the plurality of words.

4. The diagnosis step includes: A first step of extracting a plurality of words, a plurality of image frames, and a plurality of pieces of audio information included in text extracted from audio included in the reply image; a second step of deriving a plurality of first feature information, a plurality of second feature information, and a plurality of third feature information from the plurality of words, the plurality of image frames, and the plurality of speech information using respective detailed machine learning models or algorithms; and a third step of deriving derived information having a degree of depression from the plurality of first feature information, the plurality of second feature information, and the plurality of third feature information using an artificial neural network taking into account sequence data.

5. 2. The method for providing preliminary depression diagnosis information as described in claim 1, wherein the diagnostic step derives the preliminary depression diagnosis information using a machine learning model based on two or more of a plurality of words, a plurality of image frames, and audio information contained in text extracted from audio included in the reply image.

6. 5. The method for providing depression preliminary diagnosis information according to claim 4, wherein the artificial neural network considering the sequence data corresponds to a recurrent neural network or a Transformer series machine learning model based on an attention mechanism, and the first feature information, the second feature information, and the third feature information corresponding to each of the plurality of words are input in a merged form to the recurrent neural network or the Transformer series machine learning model.

7. 2. The method for providing preliminary depression diagnosis information as described in claim 1, wherein the portion of the response text displayed in the script layer is displayed with one or more of highlighting, font, size, color, and underlining changed depending on one or more of the degree of depression and the type of depression in the preliminary depression diagnosis information corresponding to the portion of the response text.

8. A part of the response text that is displayed with one or more of the highlight, font, size, color, and underline changed indicates that the degree of depression in the depression preliminary diagnosis information corresponding to the part of the response text is equal to or exceeds a predetermined criterion, 8. The method for providing depression preliminary diagnosis information of claim 7, wherein when a user selects a portion of the response text that is displayed with one or more of the highlight, font, size, color, and underline changed, the image time point of the reply image layer changes to a time point corresponding to the position of the portion of the response text.

9. The method for providing depression preliminary diagnosis information according to claim 2 , wherein the depression graph layer includes a first criterion display element indicating that the depression degree exceeds a predetermined first criterion or indicates a detailed depression type.

10. the first display screen further includes a scroll layer that displays scroll information of the script layer; The scroll layer displays information display elements at each time point displayed according to at least one of the degree of depression and the type of depression of the depression preliminary diagnosis information, 2. The method for providing preliminary depression diagnosis information according to claim 1, wherein when the user selects the information display element, the script layer moves to a position of the response text corresponding to the selected information display element.

11. The second display screen displayed in the providing step includes: a reply image layer on which the reply image is displayed; a summary script layer on which summary reply text information extracted from the reply image is displayed; the summary response text information includes one or more portions of the response text in which the degree of depression in the depression preliminary diagnosis information corresponding to the portion of the response text is equal to or greater than a predetermined level; 2. The method for providing preliminary depression diagnosis information according to claim 1, wherein the image time point of the reply image layer is changed to a time point corresponding to a position of a script portion selected by a user's input in the summary script layer.

12. the second display screen further includes one or more selection input elements corresponding to each of the summary response text information; 12. The method for providing depression preliminary diagnosis information according to claim 11, wherein a portion of a reply image corresponding to the selected input element or a portion of summary reply text information selected by a user's input is reproduced.

13. An apparatus for providing depression pre-diagnosis information using a machine learning model implemented in a computing device having one or more processors and one or more memories, comprising: a presentation unit for presenting question information to a patient terminal from the computing device; a reply image collecting unit that collects reply images including image information and voice information of responses of the patient to the question information; A diagnosis unit that derives preliminary depression diagnosis information from one or more of the returned images using a machine learning model; a providing unit for providing a user with depression preliminary diagnosis information; The first display screen displayed by the providing unit includes: a reply image layer on which the reply image is displayed; a script layer on which the question information presented to the patient terminal and reply text information extracted from the reply image are displayed, The apparatus for providing preliminary depression diagnosis information, wherein the image time point of the reply image layer changes to a time point corresponding to the position of a script portion selected by a user's input in the script layer.

14. 1. A computer program for implementing a method for providing depression pre-diagnosis information using a machine learning model executed on a computing device having one or more processors and one or more memories, comprising: On the computer, a presentation function for presenting question information to a patient terminal from the computing device; a reply image collecting function for collecting reply images including image information and audio information of responses of the patient to the question information; A diagnostic function that derives preliminary depression diagnosis information from one or more of the reply images using a machine learning model; A function of providing a user with preliminary depression diagnosis information is realized. The first display screen displayed in the providing function includes: a reply image layer on which the reply image is displayed; a script layer on which the question information presented to the patient terminal and reply text information extracted from the reply image are displayed, A computer program, wherein the image time point of the reply image layer is changed to a time point corresponding to the position of a script portion selected by user input in the script layer.

Images