A system for collecting training data, a method for collecting training data, a non-volatile storage medium, and a machine learning model.

Abstract

This invention provides a system and method for collecting training data for machine learning models in cognitive function. [Solution] The method involves acquiring training data for each of multiple test users, and collecting training data based on the acquired training data such that a first number indicating the number of training data for test users with indicators included within a predetermined neighborhood range from a boundary point is greater than a second number indicating the number of training data for users with indicators included outside the neighborhood range. The training data associates pre-obtained indicators regarding the degree of cognitive function of the test users with the test users' biometric data. The machine learning model outputs the user's indicators by inputting the user's biometric data. The biometric data indicates the user's physical activity state during or after performing a first task, and the indicators classify the user's cognitive function state into multiple classes by comparing the indicators with predetermined boundary points.

Description

Technical Field

[0001] The present invention relates to a system for collecting teacher data, a method for collecting teacher data, a non-volatile memory medium, and a machine learning model.

Background Art

[0002] Patent Document 1 discloses a computer system for estimating the state of a subject. In one embodiment, the present disclosure provides a computer system for estimating the state of a subject, the computer system including a receiving means for receiving a plurality of images of the subject walking, a generating means for generating at least one silhouette image of the subject from the plurality of images, and an estimating means for estimating at least one state related to at least one disease of the subject based on at least the at least one silhouette image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there is still room for improvement in such technologies for estimating the state of a subject.

Means for Solving the Problems

[0005] According to one aspect of the present invention, a system for collecting training data for training a machine learning model is provided, wherein the training data is configured such that an index relating to the degree of cognitive function of a test user obtained in advance is associated with the biometric data of the test user, the machine learning model is configured to output the index of the user by inputting the biometric data of the user, the biometric data indicates the physical activity state of the user during or after the user performs a first task, the index is configured to classify the state relating to the user's cognitive function into a plurality of classes by comparing the index with predetermined boundary points, and the system comprises at least one processor, the processor is configured to execute a program such that the following steps are performed: in the data acquisition step, training data for each of a plurality of test users is acquired; and in the collection step, training data is collected based on the acquired training data of the test users such that a first number indicating the number of training data for test users having an index within a predetermined neighborhood range from a boundary point is greater than a second number indicating the number of training data for users having an index outside the neighborhood range.

[0006] This configuration allows for the output of cognitive function indicators with greater accuracy. [Brief explanation of the drawing]

[0007] [Figure 1] This is a diagram illustrating an example of information processing system 1. [Figure 2] This is a block diagram showing an example of the hardware configuration of the information processing device 2. [Figure 3] This is a block diagram showing an example of the hardware configuration of medical device terminal 3. [Figure 4] This flowchart shows an example of the information processing flow performed in Information Processing System 1. [Figure 5] This is a histogram representing an example of a population dataset. [Figure 6] This is a histogram showing an example of a training dataset. [Figure 7] This figure shows recording screen 5, which is an example of a screen used to present a task. [Figure 8] An example of the evaluation results display screen is shown. [Modes for carrying out the invention]

[0008] Embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below can be combined with each other.

[0009] Incidentally, the program for implementing the software appearing in one embodiment may be provided as a non-transitory computer-readable medium, or it may be provided as a downloadable medium from an external server, or it may be provided so that the program is launched on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0010] Furthermore, in various information processing according to one embodiment, an input and an output corresponding to the input can be realized. Here, as long as an output is obtained as a result of the input, the form of the information referenced in such information processing (hereinafter referred to as "reference information") is not limited. The reference information may be, for example, rule-based information such as a database, a lookup table, or a predetermined function (including a decision formula such as a regression equation constructed by a statistical method), or a pre-trained model that has learned the correlation between input and output in advance, or a large-scale language model that can output a desired result by inputting a prompt.

[0011] Furthermore, in one embodiment, "part" may include, for example, hardware resources implemented by a circuit in a broad sense, and the information processing of software that can be specifically realized by these hardware resources. Also, in one embodiment, various types of information are handled, and this information can be represented, for example, by the physical values ​​of signal values ​​representing voltage and current, the high or low values ​​of signal values ​​as a set of binary bits composed of 0s or 1s, or by quantum superposition (so-called qubits), and communication and calculations can be performed on a circuit in a broad sense.

[0012] Furthermore, a circuit in a broad sense is a circuit realized by combining at least a suitable combination of circuits, circuits, processors, and memory. The processor may be a general-purpose processor or a dedicated circuit. In other words, it includes application-specific integrated circuits (ASICs), programmable logic devices (for example, simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc.

[0013] 1. Hardware Configuration This section describes the hardware configuration.

[0014] <Information Processing System 1> Figure 1 is a configuration diagram showing an example of an information processing system 1. The information processing system 1 comprises an information processing device 2 and a medical professional terminal 3, which is an example of a user terminal. The information processing system 1 may further include a sound collection device 4. The information processing device 2, the medical professional terminal 3, and the sound collection device 4 are configured to communicate with each other via a telecommunications line. In one embodiment, the information processing system 1 consists of one or more devices or components. For example, if the information processing system 1 consists only of the information processing device 2, then the information processing system 1 can be the information processing device 2. Similarly, if the information processing system 1 consists only of the medical professional terminal 3, then the information processing system 1 can be the medical professional terminal 3.

[0015] <Information Processing Device 2> Figure 2 is a block diagram showing an example of the hardware configuration of the information processing device 2. The information processing device 2 comprises a communication bus 20, a communication unit 21, a storage unit 22, and at least one processor 23, and these components are electrically connected within the information processing device 2 via the communication bus 20. Each component will be described further.

[0016] The communication unit 21 preferably uses wired communication methods such as USB, IEEE1394, Thunderbolt®, and wired LAN network communication, but may also include wireless LAN network communication, mobile communication such as 3G / LTE / 5G, and Bluetooth® communication as needed. In other words, it is more preferable to implement it as a collection of these multiple communication methods. That is, the information processing device 2 may communicate various information from the outside via the communication unit 21 and the network.

[0017] The storage unit 22 stores various types of information defined as described above. This can be implemented, for example, as a solid state drive (SSD), a solid state hybrid drive (SSHD), a hard disk drive (HDD), a USB (Universal Serial Bus) flash drive (USB memory), an SD memory card, a CD, a DVD, a Blu-ray (registered trademark) Disc (BD), etc., which store various programs and the like related to the information processing apparatus 2 executed by the processor 23. Or it can be implemented as a memory such as a random access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to the calculation of programs. The storage unit 22 stores various programs, variables, etc. related to the information processing apparatus 2 executed by the processor 23.

[0018] The processor 23 performs processing and control of the overall operations related to the information processing apparatus 2. The processor 23 is, for example, a central processing unit (CPU) not shown in the figure. The processor 23 realizes various functions related to the information processing apparatus 2 by reading a predetermined program stored in the storage unit 22. That is, the information processing by software stored in the storage unit 22 is specifically realized by the processor 23, which is an example of hardware, and can be executed as each functional unit included in the processor 23. Note that the processor 23 is not limited to being single, and may be implemented to have a plurality of processors 23 for each function. Or combinations thereof may also be possible.

[0019] The processor 23 is configured as an acquisition unit to acquire information from the medical staff terminal 3 or other devices. The processor 23 can acquire various information by reading out various information stored in a storage area which is at least part of the storage unit 22 and writing the read information into a work area which is at least part of the storage unit 22. The storage area is, for example, an area implemented as a storage device such as an SSD in the storage unit 22. The work area is, for example, an area implemented as a memory such as a RAM. Note that the acquisition by the processor 23 includes acquiring the output results of each functional unit included in the processor 23.

[0020] The processor 23 is configured as a display processing unit to display various information. The information can be presented to the user via the display unit 34 of the medical staff terminal 3 or other devices described later. In such a case, for example, the processor 23 controls to display visual information such as a screen, a still image or a moving image (continuous images), an icon, a message, etc. on the display unit 34 of the medical staff terminal 3. The processor 23 may generate only the rendering information for displaying the visual information on the medical staff terminal 3. Note that the processor 23 may present the output information to the user without going through the medical staff terminal 3 or other device users.

[0021] <Medical staff terminal 3> FIG. 3 is a block diagram showing an example of the hardware configuration of the medical staff terminal 3. The medical staff terminal 3 includes a communication bus 30, a communication unit 31, a storage unit 32, a processor 33, a display unit 34, and an input unit 35, and these components are electrically connected to each other inside the medical staff terminal 3 via the communication bus 30. The descriptions of the communication unit 31, the storage unit 32, and the processor 33 are omitted because they are the same as the descriptions of the respective units in the information processing apparatus 2.

[0022] The display unit 34 may be included in the medical terminal 3 housing or it may be an external component. The display unit 34 displays a graphical user interface (GUI) screen that can be operated by the user. Preferably, this is done by using different display devices such as CRT displays, liquid crystal displays, organic EL displays, and plasma displays depending on the type of medical terminal 3.

[0023] The input unit 35 is configured to accept input from the user. The input unit 35 may be included in the housing of the medical terminal 3 or it may be an external component. For example, the input unit 35 may be integrated with the display unit 34 and implemented as a touch panel. If it is a touch panel, the user can input tap operations, swipe operations, etc. Of course, instead of a touch panel, a switch button, mouse, QWERTY keyboard, voice recognition device, gesture detection device, gaze detection device, biosignal detection device, imaging device, etc. may be used. In other words, the input unit 35 accepts operation input made by the user. In response, the input unit 35 transmits a signal corresponding to the operation input to the processor 33 via the communication bus 30. The processor 33 can then perform predetermined controls and calculations as needed.

[0024] <Sound collection device 4> The sound collection device 4 is a so-called microphone configured to convert external sounds into signals. The sound collection device 4 may be provided by directly connecting the sound collection device 4 to the information processing device 2, or it may be provided with or connected to the medical worker terminal 3.

[0025] The sound collection device 4 is configured to generate voice data by collecting the user's speech. The voice data is temporarily stored in the memory of the user terminal and does not necessarily need to be stored non-volatilely in the storage unit 32. The voice data generated by the sound collection device 4 is configured to be transferable to the information processing device 2 via the network.

[0026] The sound collection device 4 is not particularly limited, but it collects sound in the human audible range, specifically between 20 Hz and 20,000 Hz, and converts it into an electrical signal. The sound may be recorded in mono or stereo. When digitally processing the sound data, the sampling rate may be, for example, 48,000 Hz, 44,100 Hz, 32,000 Hz, 22,050 Hz, 16,000 Hz, 11,025 Hz, 11,000 Hz, 8,000 Hz, etc. Any of the values ​​exemplified here may be used. By increasing the sampling rate, the temporal timing of the sound can be discretized more precisely, thereby improving the accuracy of speech recognition.

[0027] Furthermore, the data collected by the sound collection device 4 may be compressed as appropriate by the processor 33 of the medical staff terminal 3. The compression format at this time may be any of MP3, AAC, WMA, Vorbis, AC3, MP2, FLAC, TAK, etc. Compression can reduce the communication traffic caused by data transfer from the medical staff terminal 3 to the information processing device 2.

[0028] 2. Regarding information processing This section describes the information processing performed in the aforementioned information processing system 1. This information processing system 1 is, for example, a system for collecting training data for training a machine learning model through this information processing. The training data is configured so that indicators regarding the degree of cognitive function of a test user obtained in advance are associated with the test user's biometric data. The machine learning model is configured to output the indicators for the user by taking the user's biometric data as input.

[0029] Biometric data indicates the user's physical activity state during or after performing a first task. Biometric data is obtained, for example, as a result of the user's performance of the first task. The first task is a task for evaluating the user's motor and perceptual functions, which correlate with cognitive function. For example, biometric data may include at least one of the following during the performance of the first task: voice data emitted by the user, facial expression data relating to the user's facial expressions, olfactory data relating to smells perceived by the user, and motion data relating to the user's movements. Specific examples of the first task will be described later.

[0030] The indicators of cognitive function are structured to reflect the level of cognitive function defined in the clinical practice guidelines. For the sake of clarity, the indicators of cognitive function will be simply referred to as "indicators" below. The indicators are structured to classify the user's cognitive function into multiple classes by comparing it to predetermined boundary points. For example, the indicators may be structured based on scores representing the results of neuropsychological tests. The neuropsychological tests mentioned are, for example, neuropsychological tests authorized by relevant academic societies. Examples of such neuropsychological tests include the Clinical Dementia Rating (CDR), Mini-Mental State Examination (MMSE), Revised Hasegawa Dementia Scale (HDS-R), Montreal Cognitive Assessment (MoCA), Japanese version of MoCA (MoCA-J), Dementia Assessment Sheet for Community-based Integrated Care System-21 items (DASC-21), ABC dementia scale (ABC-DS), and Frontal Assessment Battery (FAB). Relevant academic societies include, for example, the Japanese Neuropsychological Society and the International Neuropsychological Association.

[0031] A cognitive function state may include, for example, a first state that may be diagnosed as dementia, a second state that may be diagnosed as mild cognitive impairment (MCI), and a third state that is other than the first and second states. For example, a score of 25 or less on the MoCA-J index suggests that the subject may have MCI, while a score of 26 or more on the MoCA-J index suggests that the subject's cognitive function is normal, i.e., the third state. Another example is that a score of 0.5 or more obtained by the CDR suggests that the subject may have MCI, while a score of 0 obtained by the CDR suggests that the subject's cognitive function is normal, i.e., the third state. The values ​​used to distinguish these states are also called cutoff values ​​and function as boundary points that allow a user's cognitive function state to be classified into multiple classes. The classes may directly represent the first, second, and third states, or they may be indirectly represented using parameters that correlate with these states. In this embodiment, a first boundary point may be included as a guideline for distinguishing between a first state and a second state, and a second boundary point may be included as a guideline for distinguishing between a second state and a third state. These boundary points may be defined as a single numerical value between two possible values ​​of the index, or they may be defined using an upper or lower limit of the index that serves as a guideline for determining which state a state belongs to. Note that the states relating to cognitive function can be arbitrarily defined as long as they include at least two types of states.

[0032] 2.1. Information Processing Flow Figure 4 is a flowchart illustrating an example of the information processing flow performed in Information Processing System 1. Note that this information processing may include arbitrary exception handling not shown. Exception handling includes interrupting the information processing or omitting individual processes. The selections or inputs performed in this information processing may be based on user operation or performed automatically without user operation.

[0033] [Step S1] First, in step S1, the processor 23 obtains the results of the test user's execution of a first task and the results of the test user's execution of a second task. The results of the first task include biometric data from the test user. The second task is to calculate an index that allows the user's cognitive function to be classified into multiple classes, and is configured, for example, to include this index as the result of the second task. Such a configuration can reduce the complexity of collecting training data. Specifically, for example, the second task is a task assigned in the neuropsychological test described above (e.g., answering a questionnaire).

[0034] [Step S2] The processor 23 constructs training data by associating the execution results of the first task with the execution results of the second task. The processor 23 then retrieves the constructed training data. At this time, the processor 23 may perform anonymization on each training data point so that the test user from which it originated cannot be identified. The method of obtaining the training data is not limited to this. For example, the processor 23 may obtain training data constructed by an external device from that external device. In other words, the processor 23 obtains the training data for multiple test users by any method.

[0035] [Step S3] Next, in step S3, the processor 23 constructs the population dataset DS1 based on the training data constructed in step S2. The population dataset DS1 consists of the training data for each of the multiple test users.

[0036] Figure 5 is a histogram representing an example of a population dataset. As shown in Figure 5, the population dataset DS1 can be represented as a distribution of the number of training data (sample size) for each index value. Each training data includes the result of the first task associated in step S2 (specifically, biometric data). For example, the dataset is configured such that the number of training data included within the neighborhood range from a certain boundary point (specifically, the first boundary point P1 or the second boundary point P2) and the number of training data included outside the neighborhood range are less than or equal to a predetermined threshold value. In Figure 5, the neighborhood range corresponding to the first boundary point P1 is shown as R1, and the neighborhood range corresponding to the second boundary point P2 is shown as R2. For example, the processor 23 may be configured such that the difference between the maximum number of training data for each index included within the neighborhood range from a certain boundary point and the minimum number of training data for each index included outside the neighborhood range is less than or equal to a threshold value. The distribution of the population dataset DS1 is arbitrary. The method of setting the neighborhood range can be appropriately set according to the estimation accuracy required for the machine learning model and the distribution required for the training dataset DS2, which will be described later. For example, the neighborhood range may be a range composed of two values ​​(25 points and 26 points in the case of MoCA-J) that indicate the boundary between two adjacent states. Furthermore, the processor 23 does not need to construct the population dataset DS1 based on the acquired training data itself; for example, it may acquire the population dataset DS1 constructed by another device.

[0037] [Step S4] Returning to Figure 4, in step S4, the processor 23 sets parameters related to the method of collecting training data. For example, the processor 23 sets a restriction parameter as such a parameter that restricts the distribution of the training dataset composed of the training data to be collected. The restriction parameter may include, for example, values ​​indicating boundary points (e.g., a first boundary point P1 and a second boundary point P2), neighborhood ranges of each boundary point (e.g., neighborhood ranges R1, R2), the number of training data for test users with indicators included within a predetermined neighborhood range from the boundary point for each boundary point (hereinafter referred to as the first number N1), the number of training data for users with indicators included outside the neighborhood range from the boundary point (hereinafter referred to as the second number N2), and the relationship between the first number N1 and the second number N2 (e.g., the ratio or tolerance value of the difference between the first number N1 and the second number N2). Here, the processor 23 sets the parameters such that N1 > N2. Such parameters are also called collection conditions for collecting training data from the population dataset DS1. Furthermore, the boundary point is not limited to a single value, but may have a certain range of values. For example, the boundary point may be defined using the upper limit of the index corresponding to one of the two states and the lower limit of the index corresponding to the other of the two states, with the range of values ​​the index can take between the upper and lower limits being used. In this case, the boundary point can be defined using the upper and lower limits.

[0038] [Step S5] Next, in step S5, the processor 23 collects training data by extracting a portion of the population dataset based on the parameters set in step S4. This allows the processor 23 to construct the training dataset DS2. For example, the processor 23 collects the training data based on the acquired test user training data such that the first number N1 is greater than the second number N2, or more specifically, the difference N1-N2 between the first number N1 and the second number N2 is greater than a baseline value. Step S5 is an example of a data collection step. The baseline value can be set appropriately based on the parameters set in step S4. With such a configuration, it is possible to facilitate the training of a machine learning model that can improve the accuracy of classifying user classes located at the boundary of cognitive function states, such as the boundary between mild cognitive impairment and dementia, or the boundary between mild cognitive impairment and healthy individuals.

[0039] Figure 6 is a histogram showing an example of a training dataset. As shown in Figure 6, the training dataset DS2 is configured such that the first number N1 is greater than the second number N2. Specifically, the training dataset DS2 is configured so that the training data is concentrated within the neighboring ranges R1 and R2. For example, in the training dataset DS2, the ratio N2 / N1 of the second number N2 to the first number N1 is, for example, 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.2, 0.21, 0.22, 0.23 ,0.24,0.25,0.26,0.27,0.28,0.29,0.3,0.31,0.32,0.33,0.34,0.35,0.36,0.37,0.38,0.39,0.4,0.41,0.42,0.43,0.44,0.45,0.46,0.47,0.48,0.49,0.5, and may be within the range of any two of the values ​​exemplified here. From the viewpoint of accuracy, the ratio N2 / N1 is preferably 0.01 or more and 0.5 or less, more preferably 0.02 or more and 0.4 or less. Even more preferably 0.05 or more and 0.3 or less. It is preferable that the ratio N2 / N1 is within the above range because it is easier to increase the accuracy of determining whether or not it is MCI and it is easier to reduce the influence of noise based on individual errors.

[0040] The number of training data to collect (e.g., total) is arbitrary, but preferably 15 or more, more preferably 50 or more, more preferably 100 or more, and even more preferably 200 or more. In other words, the processor 23 may, for example, collect at least 15 training data in total to constitute the training dataset DS2.

[0041] Thus, the training dataset DS2 is configured such that training data is extracted from an arbitrary population dataset DS1 to have a distribution that is heavily biased towards the vicinity of the boundary point. This allows the machine learning model to learn training data representing the relationship between cognitive function states and biometric data near the boundary point more heavily than training data further away from the boundary point. As a result, compared to training using a dataset composed of training data with a uniform distribution, where the sample size is the same for all indicators, it is possible to improve the accuracy of estimating cognitive function states when biometric data of a subject that is difficult to distinguish between two states near a certain boundary point is input.

[0042] [Step S6] Returning to Figure 4, in step S6, the processor 23 performs training on the machine learning model using the training data collected in step S5, i.e., the training dataset DS2. Here, the training algorithm for the machine learning model can be arbitrary, such as supervised learning or semi-supervised learning, and can be appropriately selected depending on the type of biometric data included in the training data. For example, specific machine learning algorithms include nearest neighbors, naive Bayes, decision trees, support vector machines, and deep learning using neural networks. For the sake of explanation, a machine learning model that has completed training using the training dataset DS2, which is composed of training data collected by the method described above, will be referred to as a trained model.

[0043] [Step S7] Next, in step S7, the processor 23 determines whether or not the machine learning model has been started to operate. The start of the machine learning model's operation means that the machine learning model's training is complete. The processor 23 determines whether or not the machine learning model's operation has been started, for example, based on whether or not the machine learning model's operation pipeline is running. The processor 23 may also make this determination based on whether or not the administrator of the information processing system 1 has performed an operation indicating that the operation has been started. If it is determined that the operation has not been started, the processor 23 returns to step S6 and resumes training the machine learning model. At this time, the processor 23 may return to step S5, etc., and collect the training data again.

[0044] [Step S8] On the other hand, if it is determined in step S7 that operation has started, the processor 23 determines that the training of the machine learning model is complete and restricts the updating of the training dataset and the retraining of the trained model in step S8. Determining that operation has started means, for example, that the operation pipeline of the trained model is running. As a result, the processor 23 restricts the updating of the training data included in the dataset if biometric data not associated with an metric is input after the machine learning model has been trained using the dataset composed of collected training data. With this configuration, it is possible to reduce the possibility that the consistency of the output results may be impaired by subsequent retraining of the machine learning model after the training of the machine learning model is complete and, for example, the operation of the machine learning model has started. The processor 23 may restrict this by, for example, prohibiting the execution of updating the training data, etc., but may also restrict this by displaying a warning on the display unit 34 that updates, etc., are restricted if the manager of the trained model attempts to update the training data or retrain the trained model. Note that a dataset other than the dataset used to train the trained model may be used, for example, to reconfirm the performance of the trained model, as long as the trained model is not retrained.

[0045] A machine learning model trained in this manner can also independently constitute a technical concept. Specifically, the machine learning model is trained using a pre-collected training dataset, and is configured to output an index of the subject's cognitive function level by inputting the subject's biometric data. The biometric data represents the subject's physical activity state during or after performing a first task. The index is configured to classify the subject's cognitive function state into multiple classes by comparing the index to predetermined boundary points. The training dataset is a collection of training data configured so that pre-obtained indices of the test subject's cognitive function level are associated with the test subject's biometric data. The training dataset is configured such that a first number, indicating the number of training data for test subjects with indices within a predetermined neighborhood range from the boundary points, is greater than a second number, indicating the number of training data for subjects with indices outside the neighborhood range. With such a configuration, an index of cognitive function can be obtained with high accuracy using biometric data obtained by the task, thus encouraging early detection and treatment of subjects who may have symptoms of cognitive impairment, such as mild cognitive impairment. Furthermore, a machine learning model may be configured to restrict retraining the model using a different dataset than the one used for initial training, in the event that biometric data without associated metrics is input to the model. Such a configuration reduces the possibility of inconsistency in output results being compromised by subsequent retraining of the machine learning model after it has been trained and, for example, after the model has been put into operation.

[0046] [Step S9] Next, in step S9, the processor 23 displays a screen on the display unit 34 instructing a subject, who is an example of a user, to perform the first task. By the subject performing the first task according to the instructions on the screen, the results of the first task, including the subject's biometric data, are obtained. A specific example of this screen, along with a specific example of the first task, will be described later.

[0047] [Step S10] Next, in step S10, the subject's biometric data obtained in step S9 is input into the machine learning model trained using the collected training data (i.e., the training dataset). The number of biometric data points to be input can be estimated, for example, from the difference between the theoretical value of the machine learning model and the clinically required threshold.

[0048] [Step S11] Next, in step S11, the metrics output from the machine learning model based on the input biometric data are obtained.

[0049] [Step S12] Next, in step S12, the processor 23 predicts the degree of cognitive function of a subject (e.g., at risk of dementia, at risk of MCI, normal, etc.) by outputting a class corresponding to the subject based on the acquired indicators. With this configuration, indicators related to cognitive function can be obtained with high accuracy using the biometric data obtained by the task, thus encouraging early detection and early treatment for subjects who may have symptoms of cognitive function such as mild cognitive impairment. The prediction accuracy of the machine learning model can be evaluated, for example, by comparing the predicted value with a preset threshold (e.g., confirming that the predicted value exceeds the threshold). The processor 23 may also display the prediction results on the display unit 34. At this time, the processor 23 may notify the indicators and the classes corresponding to the indicators in a manner that can be viewed at a glance. With this configuration, the interpretability of indicators related to cognitive function can be improved using the biometric data obtained by the task, and early detection and early treatment can be more effectively encouraged for subjects who may have symptoms of cognitive function such as mild cognitive impairment. The notification method can be arbitrary, such as light, sound, or vibration, but for example, the processor 23 provides notification by displaying the indicator and the corresponding class on the display unit 34 in a manner that allows for a list view.

[0050] 2.2. Specific Examples of the First Task, etc. Next, as an example of the first task, we will describe an audio task and an example of a recording screen for performing that audio task.

[0051] Figure 7 shows a recording screen 5, which is an example of a task screen for presenting a task. The recording screen 5 is an example of a screen displayed in step S9. As shown in Figure 7, the recording screen 5 includes area 51, area 52, a record button 53, a stop button 54, area 55, area 56, and area 57.

[0052] Area 51 is the area where instructions regarding speech are displayed to the subject. Area 51 displays the instruction "Tap the record button and then read the following aloud." Area 52 is the area where the specific content of the speech uttered by the subject is displayed. Area 52 displays the speech content "katama (5 times)." The method of giving the instructions regarding speech may be, for example, by voice. As mentioned above, using voice instructions makes it easier for subjects with impaired visual function due to cataracts or glaucoma to understand the instructions. Alternatively, voice instructions and display instructions may be combined. Furthermore, if the subject has glaucoma, for example, they may not be able to accurately read the instructions due to visual field defects. Therefore, the position of the display area for the instructions may be movable.

[0053] The recording button 53 is used to start recording. The processor 23 acquires audio data of the subject's speech by receiving input from the subject or medical professional to the recording button 53. The stop button 54 is used to stop recording. The processor 23 stops acquiring audio data by receiving input from the subject or medical professional to the stop button 54.

[0054] Area 55 is an area where it is visually displayed that the subject's voice is being recorded. The processor 23 displays "REC" in area 55 when it receives input from the subject or medical professional to the record button 53. The processor 23 also terminates the display of "REC" in area 55 when it receives input from the subject or medical professional to the stop button 54.

[0055] Area 56 is an area where the elapsed time since the start of recording is displayed visibly. The elapsed time may be displayed in Area 56 by object IF3. Furthermore, the elapsed time may be displayed in Area 56 by text information IF4 "00:02.00". In addition, the remaining time may be displayed visibly in Area 56.

[0056] Area 57 is an area where information indicating the timing of the subject's utterances is displayed. In other words, the processor 23 presents the subject with information including further instructions indicating the timing of utterances in a manner that can be understood. In other words, the processor 23 presents the subject with guide information including information that guides the timing of utterances in a manner that can be understood. Area 57 displays the information "katamakatama" indicating the content of what the subject will utter. Furthermore, the character information displayed in area 57 includes black-filled character information IF1 and white-outlined character information IF2. For example, in response to the timing of utterances, the character representation of the utterance content changes from white-outlined character information IF2 to black-filled character information IF1. For example, if the next word the subject utters is "ma", then in response to the timing of utterances, the white-outlined character information IF2 "ma" changes to a black-filled representation.

[0057] Here, the methods for determining the timing of speech are not limited to those described above. For example, the appearance of the characters indicating the content of the speech may change to emphasize it in response to the arrival of the timing of speech. Furthermore, the appearance of the characters may change to return to their original appearance in response to the end of the timing of speech.

[0058] Furthermore, instructions indicating the timing of speech are not limited to changes in the appearance of characters. For example, an object resembling a metronome may be displayed in area 57. The object displayed in area 57 may be presented to the subject in a manner in which the pendulum needle moves from side to side in accordance with the subject's speech timing. Alternatively, for example, an object resembling clapping hands may be displayed in area 57. The object displayed in area 57 may be presented to the subject in a manner in which the hands clap in accordance with the subject's speech timing. With such methods, the subject can speak while grasping the timing and rhythm of their speech. Therefore, it is possible to obtain audio data that reflects the subject's original speech state.

[0059] Here, we will describe an example of a method for evaluating an index indicating the degree of a subject's cognitive function from voice data. This method includes measuring and evaluating fluctuations in the speed of each individual "katama" when repeated (the so-called fixed utterance method). Furthermore, the method for evaluating the subject's index from voice data is not limited to the fixed utterance method, but may be combined with various speech patterns. It may also be combined with various evaluation methods, such as fluctuations in the movement of the feet and hips during walking. Through such a voice task, the processor 23 acquires voice data as biometric data.

[0060] 2.3. Method of notifying the results of the prediction of the level of cognitive function Next, we will explain the evaluation results display screen, which is an example of a screen used to notify users of the predicted level of cognitive function.

[0061] Figure 8 shows an example of the evaluation results display screen. The evaluation results display screen 6 shows "Cognitive Function Evaluation Results" as the evaluation target. The evaluation results display screen 6 includes area 61, area 62, area 63, and a re-measurement button 84.

[0062] Area 61 is the area where information about the subject (e.g., name, facial photograph, height, weight, BMI, etc.) is displayed.

[0063] Domain 62 is the domain where the evaluation result 621 is displayed. For example, in Domain 62, the information "Cognitive function is at a normal level" (an example of a class) is displayed as the evaluation result 621.

[0064] Region 63 is a region where the estimation results of the indicator based on the input voice data are quantitatively shown as an indicator as an evaluation result. By displaying regions 62 and 63 in a way that allows them to be viewed in a list, the indicator and the class corresponding to the indicator can be notified in a way that allows them to be viewed in a list.

[0065] This area displays a comparison result 631 showing a comparison between the current evaluation result and the previous evaluation result, if the subject has previously received the same evaluation. For example, the comparison result 631 is evaluated based on the change in the current evaluation value 634 compared to the previous evaluation value 635, which will be described later. In other words, the processor 23 further acquires evaluation information, including estimated results of the subject's past indicators. The processor 23 further evaluates the change in the degree of the subject's cognitive decline based on the evaluation information. For example, area 63 displays the information "no change" as a comparison result 631 showing a comparison with the previous time. Area 63 may also display item 832 "Current," item 833 "Previous," the current evaluation value 634, and the previous evaluation value 635 in a way that is visible to the subject. This allows the subject to understand the degree of their own cognitive function. Furthermore, it is possible to quantitatively evaluate the progress of the subject's cognitive decline. The past state may include, for example, information about the date on which the voice task was performed in the past. The processor 23 can calculate the number of days elapsed from the date of the past evaluation, etc. For example, the processor 23 may make predictions about the subject's cognitive function in the future based on the calculated past evaluation information. In this case, it is preferable that the predictions about the future progress allow for a comparison between the subject maintaining their current lifestyle and the subject changing their lifestyle. As mentioned above, if the predictions about progress when lifestyle changes are comparable, it becomes easier to maintain the subject's motivation to change their lifestyle and improve their symptoms.

[0066] Furthermore, the aforementioned lifestyle habits may include, for example, exercise habits, hobby habits, toothbrushing habits, and sleep habits (including, for example, information regarding sleep apnea syndrome).

[0067] It should be noted that Figure 8 is for illustrative purposes only and is not limited to it. While there are no restrictions on the format of the information indicating the degree of cognitive decline, it is preferable to include information that indicates stages using numbers as an indicator. More preferably, this information may be communicated as classes such as "normal," "cognitive decline," and "risk of cognitive decline" or "risk of dementia," depending on the stage.

[0068] [others] The above embodiment may be modified as follows.

[0069] The user terminal is not limited to the medical terminal 3 operated by a physician or other diagnosing person, but may also be a terminal operated by the subject themselves, or a terminal operated by the subject's caregiver, etc. The caregiver may include, for example, any person who interacts with the subject, such as a family member or acquaintance.

[0070] Users are not limited to the subjects described above, but can be any person performing the first task. Furthermore, users may include at least some of the test users.

[0071] The training dataset DS2 is not limited to those extracted from the population dataset DS1. For example, the processor 23 may construct a desired training dataset DS2 by sequentially selecting acquired training data.

[0072] The processing in steps S9 to S12 may be primarily performed by the processor 33 of the medical professional terminal 3.

[0073] The information processing device 2 may be in an on-premise configuration or a cloud configuration. In the case of a cloud-based information processing device 2, for example, the above-mentioned functions and processing may be provided in the form of SaaS (Software as a Service) or cloud computing.

[0074] In the above embodiment, the information processing device 2 performed various storage and control functions, but instead of the information processing device 2, multiple external devices may be used. That is, various information and programs may be stored in a distributed manner across multiple external devices using blockchain technology or the like.

[0075] The above embodiment is not limited to the information processing system 1, and may also be an information processing method, a program, or a non-temporary storage medium configured to store the program. The information processing method includes each step of the information processing system 1. The program causes at least one computer to execute each step of the information processing system 1.

[0076] The above-mentioned information processing system 1, etc., may be provided in any of the following embodiments.

[0077] (1) A system for collecting training data for training a machine learning model, wherein the training data is configured to associate a pre-obtained index relating to the degree of cognitive function of a test user with the biometric data of the test user, the machine learning model is configured to output the index for the user by inputting the user's biometric data, where the biometric data indicates the physical activity state of the user during or after performing a first task, the index is configured to classify the state relating to the user's cognitive function into a plurality of classes by comparing the index with predetermined boundary points, and the system comprises at least one processor, the processor is configured to execute a program so that the following steps are performed: in the data acquisition step, training data for each of the plurality of test users is acquired; and in the collection step, based on the acquired training data for the test users, the system collects the training data such that a first number indicating the number of training data for the test user having the index included within a predetermined neighborhood range from the boundary points is greater than a second number indicating the number of training data for the user having the index included outside the neighborhood range.

[0078] This configuration allows for the training of machine learning models that can improve the accuracy of classifying user classes located at the boundaries of cognitive function states, such as the boundary between mild cognitive impairment and dementia, or the boundary between mild cognitive impairment and healthy individuals.

[0079] (2) The system described in (1) above, wherein in the data acquisition step, a population dataset is acquired, which consists of the training data of each of the multiple test users, wherein the population dataset is configured such that the number of training data included in the neighborhood range and the number of training data included outside the neighborhood range are less than or equal to a predetermined reference value, and in the collection step, the training data is collected by extracting a portion of the population dataset such that the difference between the first number and the second number is greater than the reference value.

[0080] (3) The system described in (1) or (2) above, wherein in a second acquisition step, the system acquires the result of the test user performing the first task and the result of the test user performing the second task, the second task is configured such that the result of the second task includes the index, the association step constructs the training data by associating the result of the first task and the result of the second task, and the data acquisition step acquires the constructed training data.

[0081] This configuration can reduce the complexity of collecting training data.

[0082] (4) A system described in any one of (1) to (3) above, wherein the regulatory step further includes restricting the updating of the training data included in the dataset if biometric data not associated with the metric is input after the machine learning model has been trained using the dataset composed of the collected training data.

[0083] With this configuration, it is possible to reduce the possibility of inconsistency in output results being compromised by, for example, retraining the machine learning model after it has finished training and started to be used in operation.

[0084] (5) A system according to any one of (1) to (4) above, wherein the biometric data includes at least one of the following during the execution of the first task: voice data emitted by the user, facial expression data relating to the user's facial expression, olfactory data relating to a smell perceived by the user, and motion data relating to the user's movements.

[0085] (6) A system according to any one of (1) to (5) above, wherein the cognitive function state includes a first state which can be diagnosed as dementia, a second state which can be diagnosed as mild cognitive impairment, and a third state which is other than the first state and the second state, the boundary point includes a first boundary point corresponding to the boundary between the first state and the second state, and a second boundary point corresponding to the boundary between the second state and the second state, and the index is configured based on a score representing the test result in a neuropsychological test.

[0086] (7) A system according to any one of (1) to (6) above, wherein the collection step collects at least 15 of the training data.

[0087] (8) A system in which, in any one of (1) to (7) above, the input step further inputs the user's biometric data to the machine learning model trained using the training data collected in the collection step, the output acquisition step acquires the index output from the machine learning model based on the input biometric data, and the prediction step predicts the degree of cognitive function of the user by outputting a class corresponding to the user based on the acquired index.

[0088] With this configuration, indicators of cognitive function can be obtained with high accuracy using biometric data acquired through the task, thus encouraging early detection and treatment of users who may have symptoms of cognitive impairment, such as mild cognitive impairment.

[0089] (9) A system described in any one of (8) above, wherein the notification step notifies the indicator and the class corresponding to the indicator in a manner that can be viewed in a list.

[0090] With this configuration, the interpretability of indicators related to cognitive function can be improved by using biometric data obtained through the task, and it becomes possible to more effectively encourage early detection and early treatment of users who may have symptoms of cognitive impairment, such as mild cognitive impairment.

[0091] (10) A method for collecting training data for training a machine learning model, wherein the training data is configured to associate a pre-obtained index relating to the degree of cognitive function of a test user with biometric data of the test user, the machine learning model is configured to output the index of the user by inputting the user's biometric data, wherein the biometric data indicates the physical activity state of the user during or after performing a first task, and the index is configured to classify the state relating to the user's cognitive function into a plurality of classes by comparing the index with predetermined boundary points, the method comprising each step performed by the system described in any one of (1) to (9) above.

[0092] (11) A non-temporary storage medium configured to store a program for collecting training data for training a machine learning model, wherein the training data is configured to associate a pre-obtained index relating to the degree of cognitive function of a test user with the biometric data of the test user, the machine learning model is configured to output the index of the user by inputting the user's biometric data, wherein the biometric data indicates the physical activity state of the user during or after performing a first task, the index is configured to classify the state relating to the user's cognitive function into a plurality of classes by comparing the index with predetermined boundary points, and the program causes a computer to perform each step performed by the system described in any one of (1) to (9) above.

[0093] (12) A machine learning model, which is trained using a pre-collected dataset, so as to cause a computer to output an index relating to the degree of a user's cognitive function by inputting the user's biometric data, wherein the biometric data represents the user's physical activity state during or after performing a first task, the index is configured to classify the user's cognitive function state into a plurality of classes by comparing the index with predetermined boundary points, and the dataset is a set of training data configured such that a pre-obtained index relating to the degree of cognitive function of a test user is associated with the test user's biometric data, wherein a first number indicating the number of training data relating to the test user having the index included within a predetermined neighborhood range from the boundary points is greater than a second number indicating the number of training data relating to the user having the index included outside the neighborhood range.

[0094] With this configuration, indicators of cognitive function can be obtained with high accuracy using biometric data acquired through the task, thus encouraging early detection and treatment of users who may have symptoms of cognitive impairment, such as mild cognitive impairment.

[0095] (13) A machine learning model described in (12) above, wherein if the biometric data to which the indicator is not associated is input to the machine learning model, the machine learning model is configured to restrict the retraining of the machine learning model using a different dataset from the dataset used to train the machine learning model.

[0096] With this configuration, it is possible to reduce the possibility of inconsistency in output results being compromised by, for example, retraining the machine learning model after it has finished training and started to be used in operation. Of course, this is not always the case.

[0097] Finally, while various embodiments relating to this disclosure have been described, these are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]

[0098] 1: Information Processing System 2: Information Processing Device 3: Medical staff terminal 4: Sound collection device 5: Recording screen 6: Evaluation results display screen 20: Communications bus 21: Communications Department 22: Storage section 23: Processor 30: Communications bus 31: Communications Department 32: Storage section 33: Processor 34: Display section 35: Input section 51 :Area 52 :Area 53: Record button 54: Stop button 55 :Area 56 :Area 57 :Area 61 :Area 62 :Area 63 :Area 64: Remeasure button 621: Evaluation Results 631: Comparison result 634: Evaluation Value 635: Evaluation value 632 :Item 633 :Item 1000: Target Person DS1: Population dataset DS2: Training dataset IF1: Text information IF2: Text information IF3: Object IF4: Text information N1: The first number N2: The second number P1: First boundary point P2: Second boundary point R1: Neighborhood range R2: Neighborhood range

Claims

1. A system for collecting training data for training machine learning models, The aforementioned training data is configured such that an index of the degree of cognitive function of the test user obtained in advance is associated with the biometric data of the test user. The machine learning model is configured to output the aforementioned indicators for a user by inputting the user's biometric data, and here, The aforementioned biometric data indicates the user's physical activity state during or after the user performs the first task. The aforementioned index is configured to classify the state of the user's cognitive function into multiple classes by comparing the index with predetermined boundary points. The system comprises at least one processor, the processor configured to execute a program such that the following steps are performed: In the data acquisition step, the training data for each of the multiple test users is acquired. A system that, in the collection step, collects training data based on the acquired training data of the test users, such that the ratio of the number of training data for test users having the index that are included within a predetermined neighborhood range from the boundary point to the number of training data for users having the index that are included outside the neighborhood range, i.e., the ratio of the number of second training data (N2) / the number of first-rank training data (N1), is 0.01 or more and 0.8 or less.

2. In the system described in claim 1, In the data acquisition step, a population dataset is acquired, which consists of the training data of each of the multiple test users, wherein the population dataset is configured such that the number of training data included within the neighborhood range and the number of training data included outside the neighborhood range are less than or equal to a predetermined threshold value. The system, in the collection step, collects training data by extracting a portion of the population dataset such that the difference between the first number and the second number is greater than the reference value.

3. In the system described in claim 1, Furthermore, in the second acquisition step, the results of the first task performed by the test user and the results of the second task performed by the test user are acquired, and the second task is configured such that the results of the second task include the aforementioned indicator. In the association step, the training data is constructed by associating the results of the first task with the results of the second task. The data acquisition step involves acquiring the constructed training data.

4. In the system described in claim 1, Furthermore, in the regulatory step, the system restricts the updating of the training data included in the dataset if, after the machine learning model has been trained using the dataset composed of the collected training data, biometric data not associated with the metric is input.

5. In the system described in claim 1, The system includes, for the biological data, at least one of the following: voice data emitted by the user during the execution of the first task; facial expression data relating to the user's facial expressions; olfactory data relating to smells perceived by the user; and motion data relating to movements associated with the user's movements.

6. In the system described in claim 1, The aforementioned cognitive function states include a first state which may be diagnosed as dementia, a second state which may be diagnosed as mild cognitive impairment, and a third state which is other than the first and second states. The boundary point includes a first boundary point corresponding to the boundary between the first state and the second state, and a second boundary point corresponding to the boundary between the second state and the second state. The aforementioned index is a system based on scores representing test results in neuropsychological examinations.

7. In the system described in claim 1, The system collects at least 15 of the aforementioned training data in the collection step.

8. In the system described in claim 1, Furthermore, in the input step, the user's biometric data is input to the machine learning model that was trained using the training data collected in the collection step. In the output acquisition step, the indicators output from the machine learning model based on the input biometric data are acquired. In the prediction step, the system predicts the degree of cognitive function of a user by outputting a class corresponding to the user based on the acquired indicators.

9. In the system described in claim 8, In the notification step, the system notifies the indicator and the class corresponding to the indicator in a listable manner.

10. A method for collecting training data for training machine learning models, The aforementioned training data is configured such that an index of the degree of cognitive function of the test user obtained in advance is associated with the biometric data of the test user. The machine learning model is configured to output the aforementioned indicators for a user by inputting the user's biometric data, and here, The aforementioned biometric data indicates the user's physical activity state during or after the user performs the first task. The aforementioned index is configured to classify the state of the user's cognitive function into multiple classes by comparing the index with predetermined boundary points. The method comprises each step performed by the system described in any one of claims 1 to 9.

11. Non-temporary storage medium, It is configured to store a program for collecting training data for training machine learning models. The aforementioned training data is configured such that an index of the degree of cognitive function of the test user obtained in advance is associated with the biometric data of the test user. The machine learning model is configured to output the aforementioned indicators for a user by inputting the user's biometric data, and here, The aforementioned biometric data indicates the user's physical activity state during or after the user performs the first task. The aforementioned index is configured to classify the state of the user's cognitive function into multiple classes by comparing the index with predetermined boundary points. The program is a non-temporary storage medium that causes a computer to perform each step performed by the system described in any one of claims 1 to 9.

12. It is a machine learning model, By being trained using a pre-collected dataset, the computer is configured to output an index regarding the degree of the user's cognitive function when the user's biometric data is input, and here, The aforementioned biometric data indicates the user's physical activity state during or after the user performs the first task. The aforementioned index is configured to classify the state of the user's cognitive function into multiple classes by comparing the index with predetermined boundary points. The aforementioned dataset is A set of training data configured such that an index of the degree of cognitive function of a test user obtained in advance is associated with the biometric data of the test user. A machine learning model configured such that a first number indicating the number of training data for test users having the index that are included within a predetermined neighborhood range from the boundary point is greater than a second number indicating the number of training data for users having the index that are included outside the neighborhood range.

13. In the machine learning model described in claim 12, A machine learning model configured such that, when the biometric data to which the aforementioned indicator is not associated is input to the machine learning model, it is restricted from retraining the machine learning model using a different dataset than the one used to train the machine learning model.

Images