Information processing device, information processing method, and information processing program

Abstract

To provide a technology for advantageously evaluating a growth of a newborn baby based on biological information of a subject user which can be relatively easily detected.SOLUTION: An information processing device of the present disclosure evaluates a growth of a subject user including a new born baby. The information processing device includes a control part which: obtains user biological information which is predefined biological information of a subject user detected by predefined detection means; obtains active state information which is information related to an active state of the subject user including the predefined state amount of the subject user and the information related to a physical activity based on the user biological information; determines active state change patterns which is a change pattern of the active state shown by the subject user based on the active state information; and evaluates a growth of a subject user based on the active state change pattern.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program for evaluating the development of a subject user, including a newborn baby. [Background technology]

[0002] In recent years, it has been said that many people, including children, are suffering from behavioral disorders, developmental delays, and neurological dysfunction. Therefore, by detecting developmental disorders early, for example through interviews at the 18-month health checkup, symptoms can be alleviated through therapeutic education and the child's ability to adapt to society can be improved.

[0003] For example, it has been revealed that infants with autism have abnormalities in the distribution of their gaze points, and it is believed that such symptoms appear at an extremely early stage. Therefore, technology has been developed to evaluate the possibility of developmental disorders based on the subjects' gaze points.

[0004] For example, Patent Document 1 discloses an evaluation device that detects position data of a subject's gaze point based on image data of the subject's eyeballs and evaluates whether the subject may have a developmental disorder. This evaluation device can evaluate the subject's possibility of a developmental disorder based on the result of determining whether the subject is gazing at a natural image or a geometric image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-166254 Summary of the Invention [Problem to be solved by the invention]

[0006] It is generally difficult for parents to determine whether their newborn baby is close to typical development or whether it may have a developmental disorder. On the other hand, experts (specialists) can make such judgments based on their extensive experience, but there are problems such as a shortage of experts (specialists) with such experience and the time it takes to conduct a medical interview.

[0007] According to the technology described in Patent Document 1, evaluation images including a natural image and a geometric image arranged to surround at least a portion of the natural image are displayed on a display screen, and by determining whether the subject is gazing at the natural image or the geometric image, the possibility of a developmental disorder in the subject can be evaluated. This appears to enable early detection of developmental disorders without the need for a medical interview by a specialist. However, this technology requires a display that displays the evaluation images for the subject to view in order to evaluate the subject's development. Furthermore, it also requires, for example, a lighting device that irradiates the eyeball with near-infrared light and a stereo camera that captures the eyeball irradiated with near-infrared light. In other words, a dedicated device is required to detect position data of the subject's gaze point based on the evaluation images. Furthermore, applying such a dedicated device to test users such as newborns can be difficult, which may result in inaccurate information for evaluating development.

[0008] An object of the present disclosure is to provide a technology that can suitably evaluate the development of a newborn baby based on biological information of a subject user that can be detected relatively easily. [Means for solving the problem]

[0009] The information processing device disclosed herein is an information processing device for evaluating the development of a subject user, including a newborn. The information processing device includes a control unit that executes the following operations: acquires user biometric information, which is predetermined biometric information of the subject user detected by a predetermined detection means; acquires activity status information, which is information about the subject user's activity status, including predetermined state quantities and information about the subject user's physical activity, based on the user biometric information; determines an activity status change pattern, which is a change pattern in the activity status exhibited by the subject user, based on the activity status information; and evaluates the subject user's development based on the activity status change pattern. The control unit acquires a state based on the Brazelton Newborn Behavioral Rating Scale, using coordinate information of the subject user's body parts and / or coordinate information of the subject user's facial parts and / or information about the subject user's vocalizations, as the state quantities of the subject user; and acquires information about the movement states of the subject user's limbs, head, and trunk as information about the subject user's physical activity.

[0010] The information processing device described above can easily acquire information about the activity state of a subject user without using a display that displays an evaluation image for the subject user to view or preparing a dedicated device to evaluate the subject user's development. Furthermore, by evaluating the subject user's development based on the pattern of changes in the subject user's activity state, the subject user's development can be evaluated relatively accurately. In other words, according to the present disclosure, the development of a newborn baby can be suitably evaluated based on the subject user's biological information, which can be detected relatively easily.

[0011] Here, in the information processing device, when acquiring the state quantity of the test user, the control unit may acquire the state based on the eye open state of the test user, which is based on the vertical distance between the test user's eyes. Alternatively, when acquiring the state quantity of the test user, the control unit may acquire the state based on the emotional state of the test user, which is based on the vertical and horizontal distance between the test user's mouth, or / and the vertical and horizontal distance between the test user's eyebrows, or / and the vertical and horizontal distance between the test user's eyes. Alternatively, when acquiring the state quantity of the test user, the control unit may acquire the state based on the frequency band of the test user's crying.

[0012] In the developmental evaluation of the subject user according to the present disclosure, for example, the effectiveness of a medical treatment on the subject user may be evaluated. In this case, the control unit can determine a pain onset period as the activity state change pattern based on the alertness, sedation, vocalization, body movement, facial tension, and muscle tension of the subject user, and evaluate the effectiveness of the medical treatment on the subject user based on the activity state change pattern.

[0013] In addition, in the information processing device of the present disclosure, the control unit may determine the activity state change pattern by inputting the user's biometric information into a pre-learning model constructed by a neural network model having an input layer that accepts input of specified input image data, an intermediate layer that extracts features representing information related to the activity state of the subject user from the input image data, and an output layer that outputs an identification result based on the features.

[0014] The present disclosure can also be viewed from the perspective of an information processing method by a computer. That is, the information processing method of the present disclosure is an information processing method for evaluating the development of a subject user, including a newborn, in which a computer executes the following steps: acquires user biological information, which is predetermined biological information of the subject user detected by a predetermined detection means; acquires activity state information, which is information on the activity state of the subject user, including predetermined state quantities and information on physical activity of the subject user based on the user biological information; determines an activity state change pattern, which is a change pattern in the activity state exhibited by the subject user, based on the activity state information; evaluates the development of the subject user based on the activity state change pattern; acquires a state based on the Brazelton Newborn Behavioral Rating Scale, using coordinate information of the subject user's body parts and / or coordinate information of the subject user's face parts and / or information on vocalizations by the subject user, as the state quantities of the subject user; and acquires information on the movement states of the subject user's limbs, head, and trunk as information on the subject user's physical activity.

[0015] The present disclosure can also be understood from the aspect of an information processing program. That is, the information processing program of the present disclosure is an information processing program for evaluating the development of a subject user, including a newborn, and causes a computer to execute the following operations: acquire user biological information, which is predetermined biological information of the subject user detected by a predetermined detection means; acquire activity state information, which is information on the activity state of the subject user, including predetermined state quantities and information on physical activity of the subject user based on the user biological information; determine an activity state change pattern, which is a change pattern in the activity state exhibited by the subject user, based on the activity state information; evaluate the development of the subject user based on the activity state change pattern; acquire a state based on the Brazelton Newborn Behavioral Rating Scale, using coordinate information of the subject user's body parts and / or coordinate information of the subject user's face parts and / or information on vocalizations by the subject user, as the state quantities of the subject user; and acquire information on the motor states of the subject user's limbs, head, and trunk as information on the physical activity of the subject user. [Effects of the Invention]

[0016] According to the present disclosure, the development of a newborn baby can be suitably evaluated based on biological information of a subject user that can be detected relatively easily. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a diagram illustrating a schematic configuration of an information processing system according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing in more detail the components of a server and a user terminal included in the information processing system in the first embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of the flow of operations of the information processing system in the first embodiment. [Figure 4] FIG. 10 is a diagram for explaining detection of user biometric information using a user terminal. [Figure 5]FIG. 2 is a diagram showing states in the Brazelton Newborn Behavioral Rating Scale as state quantities of a subject user in the first embodiment. [Figure 6] FIG. 10 is a diagram for explaining the classification result obtained from an input to a pre-training model in the second embodiment, and the neural network that constitutes the pre-training model. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. The configurations of the following embodiments are examples, and the present disclosure is not limited to the configurations of the embodiments.

[0019] First Embodiment An overview of an information processing system in a first embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing a schematic configuration of the information processing system in this embodiment. The information processing system 100 according to this embodiment includes a network 200, a server 300, and a user terminal 400. The information processing system 100 in this embodiment is a system for evaluating the development of a subject user, and its processing is executed by the server 300. Furthermore, users who use the information processing system 100 in this embodiment are subject users including newborns and assistant users who assist the subject users in measuring their biological information, etc. For example, the assistant users are staff members of a predetermined facility (such as a hospital). In this embodiment, an example of predicting the developmental prognosis of a subject user as a developmental evaluation of the subject user will be described below.

[0020] Network 200 is, for example, an IP network that connects server 300 and user terminal 400 so that they can communicate with each other. As long as network 200 is an IP network, it may be wireless, wired, or a combination of wireless and wired. For example, in the case of wireless communication, user terminal 400 may access a wireless LAN access point (not shown) and communicate with server 300 via a LAN or WAN. Furthermore, network 200 is not limited to these examples and may be, for example, a public switched telephone network, an optical fiber line, an ADSL line, a satellite communication network, or a network related to short-range communication such as Bluetooth (registered trademark).

[0021] The server 300 is connected to a user terminal 400 via a network 200. For ease of explanation, one server 300 and four user terminals 400 are shown in Fig. 1, but it goes without saying that the number of user terminals is not limited to this.

[0022] Server 300 may be any electronic device having the processing power for arithmetic and processing operations such as data acquisition, generation, and updating, including personal computers, servers, mainframes, and other electronic devices. That is, server 300 may be configured as a computer having a processor such as a CPU or GPU, a main memory such as RAM or ROM, and an auxiliary memory such as an EPROM, a hard disk drive, or removable media. The removable media may be, for example, a USB memory or a disk recording medium such as a CD or DVD. The auxiliary memory stores an operating system (OS), various programs, various tables, and the like.

[0023] In addition, the server 300 may appropriately use SaaS (Software as a Service), Paas (Platform as a Service), or IaaS (Infrastructure as a Service) using a cloud server, without providing software, hardware, an OS, etc. dedicated to the information processing system 100 of this embodiment.

[0024] The user terminal 400 may be any electronic device such as a mobile terminal owned by a user (in this embodiment, an assistant user) who uses the information processing system 100, and may be, for example, a mobile terminal, a tablet terminal, a smartphone, a wearable terminal, a personal computer, or other terminal device.

[0025] Next, the components of the server 300 and the user terminal 400 will be mainly described in detail with reference to Fig. 2. Fig. 2 is a diagram showing in more detail the components of the server 300 and the user terminal 400 included in the information processing system 100 in the first embodiment.

[0026] The server 300 has, as functional units, a communication unit 301, a storage unit 302, and a control unit 303. The server 300 loads a program stored in an auxiliary storage device into a working area of ​​a main storage device and executes it. The execution of the program controls each functional unit, thereby realizing each function that matches the predetermined purpose of each functional unit. However, some or all of the functions may be realized by hardware circuits such as ASICs and FPGAs.

[0027] Here, the communication unit 301 is a communication interface for connecting the server 300 to the network 200. The communication unit 301 is configured to include, for example, a network interface board and a wireless communication circuit for wireless communication. The server 300 is connected to the user terminal 400 and other external devices via the communication unit 301 so as to be able to communicate with them.

[0028] The storage unit 302 includes a main storage unit and an auxiliary storage unit. The main storage unit is a memory in which programs executed by the control unit 303 and data used by the control programs are developed. The auxiliary storage unit is a device in which programs executed by the control unit 303 and data used by the control programs are stored. The storage unit 302 stores an evaluation scale, which will be described later, in advance. The storage unit 302 also stores data transmitted from the user terminal 400, etc., and the storage unit 302 can store user biometric information, which will be described later. The server 300 can acquire data transmitted from the user terminal 400, etc., via the communication unit 301.

[0029] The control unit 303 is a functional unit that controls the server 300. The control unit 303 can be realized by an arithmetic processing unit such as a CPU. The control unit 303 further includes four functional units: an acquisition unit 3031, a calculation unit 3032, a determination unit 3033, and an evaluation unit 3034. Each functional unit may be realized by the CPU executing a stored program.

[0030] The acquiring unit 3031 acquires user biometric information of the subject user. Here, the user biometric information is predetermined biometric information of the subject user, such as information about the movements of the subject user's body parts (limbs, head, trunk, etc.) and facial parts (eyes, eyebrows, mouth, etc.), and vocalizations (crying, etc.). The assisting user can detect these using the user terminal 400, which corresponds to the detecting means of the present disclosure, and transmit them to the server 300. The acquiring unit 3031 acquires the user biometric information by acquiring such transmitted information, and stores it in the storage unit 302 of the server 300.

[0031] Here, the user terminal 400 in this embodiment has, as functional units, a communication unit 401, an input / output unit 402, and a storage unit 403. The communication unit 401 is a communication interface for connecting the user terminal 400 to the network 200, and is configured to include, for example, a network interface board and a wireless communication circuit for wireless communication. The input / output unit 402 is a functional unit for displaying information transmitted from the outside via the communication unit 401, and for inputting information when transmitting the information to the outside via the communication unit 401. The storage unit 403 is configured to include a main storage device and an auxiliary storage device, similar to the storage unit 302 of the server 300.

[0032] The input / output unit 402 further includes a display unit 4021, an operation input unit 4022, and an image / audio input / output unit 4023. The display unit 4021 has a function of displaying various information and is realized, for example, by an LCD (Liquid Crystal Display) display, an LED (Light Emitting Diode) display, an OLED (Organic Light Emitting Diode) display, or the like. The operation input unit 4022 has a function of accepting operation input from a user and is realized, specifically, by soft keys or hard keys on a touch panel or the like. The image / audio input / output unit 4023 has a function of accepting input of images such as still images and videos and is realized, specifically, by a camera using an image sensor such as a Charged-Coupled Device (CCD), a Metal-Oxide-Semiconductor (MOS), or a Complementary Metal-Oxide-Semiconductor (CMOS). The image / audio input / output unit 4023 has a function of accepting input and output of audio and is realized, specifically, by a microphone or speaker.

[0033] The assistant user can use the user terminal 400 configured in this manner to transmit the user biometric information to the server 300. Here, the server 300 may provide the user terminal 400 with an interface for inputting the user biometric information. In this case, the assistant user can transmit the user biometric information to the server 300 by inputting information into the interface via the user terminal 400.

[0034] The calculation unit 3032 calculates activity state information of the test user based on the user's biological information. Here, the activity state information is information related to the test user's activity state, and includes information related to the test user's predetermined state quantity and physical activity. The calculation unit 3032 calculates a state based on a predetermined evaluation scale using, as the state quantity, coordinate information of the test user's body parts, or / and coordinate information of the test user's facial parts, or / and information related to the test user's vocalizations. Furthermore, the calculation unit 3032 acquires information related to the movement states of the test user's limbs, head, and trunk as the information related to the physical activity.

[0035] The state quantities may include concepts such as the test user's alertness, activity level, sedation level, and medical treatment effect scale. The evaluation scale may be, for example, the Brazelton Neonatal Behavioral Assessment Scale (NBAS). In the present embodiment, an example of evaluating a test user's developmental delay based on the Brazelton Neonatal Behavioral Assessment Scale (NBAS) will be described. However, in the developmental evaluation of a test user in the present disclosure, the test user's sedation level may be evaluated using, for example, the following evaluation scales. In this case, for example, in evaluating the test user's pain, the ABC pain scale (ABC), Acute Pain In Newborns (APN / DAN), Behavloral Indicators of Infant Pain (BIIP), etc. may be used for acute pain, and the EVENDOL behavioral pain scale (EVENDOL), Neonatal Facial Coding Scale (NFCS), etc. may be used for chronic pain. Additionally, for example, evaluation scales such as COMFORTneo and the Neonatal Pain, Agltation and Sedation Scale (N-PASS) may be used to evaluate the sedation of a subject user.

[0036] The determination unit 3033 determines the activity state change pattern of the test user based on the activity state information. Here, the activity state change pattern represents a change pattern of the activity state exhibited by the test user. Details of the processing performed by the determination unit 3033 will be described later with reference to FIG. 3.

[0037] The evaluation unit 3034 evaluates the development of the subject user based on the above-mentioned activity state change pattern. Details of the processing performed by the evaluation unit 3034 will be explained later with reference to FIG.

[0038] The control unit 303 executes the processes of the acquisition unit 3031, calculation unit 3032, determination unit 3033, and evaluation unit 3034, thereby functioning as a control unit according to the present disclosure.

[0039] Here, the flow of operations of the information processing system 100 in this embodiment will be described. Fig. 3 is a diagram illustrating the flow of operations of the information processing system 100 in this embodiment. Fig. 3 explains the flow of operations between the components of the information processing system 100 in this embodiment and the processes executed by each component. Note that the example shown in Fig. 3 explains an aspect in which an activity state change pattern is determined using states based on the Brazelton neonatal behavioral assessment.

[0040] In the example shown in FIG. 3 , first, the user terminal 400 of the assistant user detects user biometric information, which is biometric information of a newborn baby who is a test user (S101). The assistant user can detect the user biometric information using the input / output unit 402 of the user terminal 400. Here, FIG. 4 is a diagram for explaining detection of user biometric information using the user terminal 400. The screen SC1 shown in FIG. 4 is displayed on the display unit 4021 of the user terminal 400. The screen SC1 shown in FIG. 4( a) displays a comment SC11 from the system and a detection field SC12. FIG. 4 shows an example of detecting the eye movement of the test user. The assistant user can detect the eye movement of the test user by using the camera of the user terminal 400 as a detection means and pointing the camera at the test user's eyes so that the test user's eyes are displayed in the detection field SC12. Then, when the eye movement of the test user is detected, the screen displayed on the display unit 4021 of the user terminal 400 transitions to the screen SC1 shown in FIG. 4( b). 4(b) displays a comment SC13 from the system, a send button SC14, and a cancel button SC15. The assistant user can send the detected user biometric information to the server 300 by pressing the send button SC14 using the operation input unit 4022.

[0041] It should be noted that the detection of user biometric information in this disclosure is not intended to be limited to an example using a dedicated interface such as that shown in FIG. 4, and for example, the eye movements of the subject user may be detected using well-known image recognition processing based on a facial photograph of the subject user taken by the assisting user using the camera of the user terminal 400.

[0042] In addition, the detection of user biometric information in the processing of S101 may include not only the eye movements of the test user as described above, but also the mouth movements and eyebrow movements of the test user, as well as the crying of the test user detected using the microphone of the user terminal 400 as a detection means.

[0043] Furthermore, in the process of S101, the movements of the subject user's body parts (such as the limbs, head, and trunk) are detected as user biometric information. In this case as well, the camera of the user terminal 400 is used as a detection means, and the movements of the subject user's body parts can be detected by well-known image recognition processing based on a photograph of the subject user's body taken by the assistant user.

[0044] 3, the server 300 acquires the user biometric information transmitted from the user terminal 400 (S102). The server 300 then stores the acquired user biometric information in the storage unit 302.

[0045] Next, the server 300 calculates the state based on the Brazelton newborn behavioral assessment using the coordinate information of the body parts of the test user, or / and the coordinate information of the face parts of the test user, or / and the information about the vocalizations of the test user as the state quantity of the test user (S103). This will be described below.

[0046] 5 is a diagram showing states in the Brazelton Newborn Behavioral Rating Scale as state quantities of a test user in this embodiment. As shown in FIG. 5, the Brazelton Newborn Behavioral Rating Scale is divided into six states: State 1, non-REM sleep (deep sleep), State 2, REM sleep (light sleep), State 3, drowsy state, State 4, alert state, State 5, active state, and State 6, crying state.

[0047] In this embodiment, when acquiring the state quantity of the subject user, the state is acquired based on the eye open state of the subject user, which is determined based on the vertical distance between the subject user's eyes, and the vertical and horizontal distance between the subject user's mouth. At this time, the server 300 calculates the state quantity of the subject user as State 1 to State 3, for example, when the subject user's upper eyelid and lower eyelid are overlapping, based on the coordinate information of the subject user's eyes acquired as user biometric information. In this case, specifically, when the subject user's eyelids are not moving slightly, the state quantity of the subject user is calculated as State 1. When the subject user's mouth is moving periodically, the state quantity of the subject user is calculated as State 2, based on the vertical and horizontal distance between the subject user's mouth. Furthermore, when the subject user's upper eyelid and lower eyelid are intermittently overlapping, the state quantity of the subject user is calculated as State 3.

[0048] On the other hand, based on the positions of the upper and lower eyelids of the test user, when the test user has their eyes open, the server 300 calculates the state quantity of the test user as State 4 to State 6. In this case, more specifically, based on the vertical and horizontal distances of the test user's mouth, or / and the vertical and horizontal distances of the test user's eyebrows, or / and the vertical and horizontal distances of the test user's eyes, when it is estimated that the test user's facial expression changes significantly and the test user's emotional state is not comfortable, the server 300 calculates the state quantity of the test user as State 5. In this state, further, when the frequency band of the test user's crying is 3000 to 4000 Hz, the server 300 calculates the state quantity of the test user as State 6.

[0049] 3, the server 300 calculates the amount of physical activity of the test user (S104). Here, the amount of physical activity is an amount of activity that can be calculated based on information on the motion state of the test user's limbs, head, and trunk, and, for example, the amount of activity per unit time can be calculated based on the speed and amount of movement of the test user's body parts (limbs, head, trunk, etc.).

[0050] Next, the server 300 determines the activity state change pattern of the subject user based on the state calculated in the process of S103 and the physical activity calculated in the process of S104 (S105).

[0051] Then, the server 300 evaluates the development of the subject user based on the activity state change pattern determined in the process of S105 (S106).

[0052] Then, the server 300 transmits the information regarding the development of the test user to the user terminal 400 of the assistant user. The user terminal 400 of the assistant user then acquires the information (S107). This allows the assistant user to understand the developmental state of the test user.

[0053] According to this method of evaluating the subject user's development, there is no need to use a display that displays evaluation images for the subject user to view, or to prepare a dedicated device, in order to evaluate the subject user's development.The subject user's development can be evaluated relatively accurately based on image data and audio data detected using the supporting user's user terminal 400.

[0054] According to the information processing system 100 described above, the development of a newborn baby can be suitably evaluated based on biological information of the subject user, which can be detected relatively easily.

[0055] Second Embodiment The information processing system 100 according to the second embodiment will be described below. In this embodiment, the server 300 inputs user biometric information into a pre-learning model to determine the activity state change pattern of the subject user. Here, the pre-learning model may be constructed using a neural network model having an input layer that accepts input of predetermined input image data, an intermediate layer that extracts features representing information related to the subject user's activity state from the input image data, and an output layer that outputs a classification result based on the features. Note that such a pre-learning model may be a supervised machine learning model or an unsupervised machine learning model (such as one constructed by clustering, anomaly detection, or self-supervised learning). In this embodiment, an example using a supervised machine learning model will be described.

[0056] FIG. 6 illustrates the classification results obtained from input to a pre-training model in this embodiment and the neural network that constitutes the pre-training model. In this embodiment, a neural network model generated by deep learning is used as the pre-training model. The pre-training model 30 in this embodiment includes an input layer 31 that receives input of predetermined input image data, an intermediate layer (hidden layer) 32 that extracts features representing information about the subject user's activity state from the input image data input to the input layer 31, and an output layer 33 that outputs a classification result based on the features. In the example of FIG. 6, the pre-training model 30 includes one intermediate layer 32, with the output of the input layer 31 input to the intermediate layer 32 and the output of the intermediate layer 32 input to the output layer 33. However, the number of intermediate layers 32 does not need to be limited to one; the pre-training model 30 may include two or more intermediate layers 32.

[0057] 6, each of the layers 31 to 33 includes one or more neurons. For example, the number of neurons in the input layer 31 can be set according to the input image data. The number of neurons in the output layer 33 can be set according to the activity state change pattern of the subject user, which is the classification result.

[0058] Neurons in adjacent layers are then connected as appropriate, and weights (connection loads) are set for each connection based on the results of machine learning. In the example of Figure 6, each neuron is connected to all neurons in the adjacent layer, but the neuron connections are not limited to this example and can be set as appropriate.

[0059] Such a pre-training model 30 is constructed by performing supervised learning using training data, which is a combination of multiple image data representing the activity states of a newborn baby and labels of the activity states. Specifically, the combination of feature values ​​and labels is provided to a neural network, and the weights of the connections between neurons are tuned so that the output of the neural network is the same as the label. In this way, the features of the training data are learned and a pre-training model for estimating results from inputs is inductively acquired.

[0060] Specifically, the image recognition deep learning model is trained using training data that is a combination of image data of newborns with their eyes open and closed and labels of their activity states, an image recognition deep learning model trained using training data that is a combination of image data of newborns' crying and non-crying merospectrum and labels of their activity states, an image recognition deep learning model trained using training data that is a combination of image data of newborns' facial expressions and labels of their activity states, or an image recognition deep learning model trained using images labeled by experts (doctors, nurses, etc.).

[0061] The information processing system 100 described above also makes it possible to appropriately evaluate the development of a newborn baby based on biological information of a subject user that can be detected relatively easily.

[0062] <Other variations> The above-described embodiment is merely an example, and the present disclosure may be modified as appropriate within the scope of the present disclosure. For example, the processes and means described in the present disclosure may be freely combined as long as no technical contradiction occurs.

[0063] In the above embodiment, an example of evaluating a developmental delay of a subject user has been described, but in the developmental evaluation of a subject user of the present disclosure, an estimation of the effectiveness of a medical treatment on the subject user may also be performed.

[0064] For example, the server 300 can determine the onset of pain as an activity state change pattern based on the test user's alertness, sedation, vocalization, body movement, facial tension, and muscle tension, and evaluate the effectiveness of medical treatment on the test user based on this activity state change pattern. In this case, the server 300 can acquire states for each of the six evaluation indexes (alertness, sedation, vocalization, body movement, facial tension, and muscle tension) and evaluate the effectiveness of medical treatment on the test user based on these states.

[0065] Specifically, the server 300 assigns a score of 1 to 5 to each state of the six evaluation indexes based on the activity status information of the test user for any two-minute period. If the total score for each state is 14 points or higher, the server 300 can determine that the test user is experiencing pain. Based on this determination result, a medical professional can then begin medical treatment for the test user. Furthermore, if the total score for each state falls below 9 points as a result of the medical treatment, the server 300 can determine that the medical treatment was effective. This allows the medical professional to take further measures, such as reducing the amount of sedative administered to the test user.

[0066] For example, the server 300 assigns a score to the test user's level of alertness: 1 point for closed eyes with no facial movement, 2 points for closed eyes with facial movement, 3 points for open eyes with no facial movement, 4 points for open eyes with facial movement, and 5 points for warning movements. For example, the server 300 assigns a score to the test user's level of sedation: 1 point for a calm state, 2 points for a slight anxiety state, 3 points for a controllable excited state, 4 points for an uncontrollable excited state, and 5 points for an uncontrollable distress state. For example, the server 300 assigns a score to the test user's vocalizations: 1 point for not crying, 2 points for a faint cry, 3 points for a moan, 4 points for a loud cry, and 5 points for a scream. For example, the test user's body movements are scored as follows: no movement is scored as 1 point, up to three slight arm or leg movements are scored as 2 points, three or more slight arm or leg movements are scored as 3 points, up to three violent arm or leg movements are scored as 4 points, and three or more violent arm or leg movements are scored as 5 points. For example, the test user's facial tension is scored as follows: 1 point for completely relaxed with the mouth open, 2 points for normal tension, 3 points for intermittent eye squinting and frown, 4 points for continuous eye squinting and frown, and 5 points for frowns with muscles distorted. For example, the test user's muscle tension (muscle tension in the body) is scored as follows: 1 point for completely relaxed with the arms spread, 2 points for less tension than normal, 3 points for normal tension, 4 points for clenched hands and increased tension, and 5 points for excessive muscle tension with stiff muscles.

[0067] In addition, the server 300 may determine the effectiveness of medical treatment for the subject user by inputting user biometric information into a pre-trained model trained using training data that is a combination of audio and image data defined for the above six evaluation indicators and labels for evaluating the effectiveness of medical treatment.

[0068] In addition, in the above embodiment, an example was described in which the processes of the acquisition unit 3031, calculation unit 3032, judgment unit 3033, and evaluation unit 3034 are executed by the server 300, but these processes may also be executed by the user terminal 400.

[0069] Furthermore, the processing described as being performed by one device may be shared and executed by a plurality of devices. For example, the acquisition unit 3031 may be formed in a processing device separate from the server 300. In this case, the separate processing device is configured to be able to cooperate favorably with the server 300. Furthermore, the processing described as being performed by different devices may be executed by a single device. In a computer system, the hardware configuration (server configuration) by which each function is realized can be flexibly changed.

[0070] The present disclosure can also be realized by providing a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors in the computer read and execute the program. Such a computer program may be provided to the computer via a non-transitory computer-readable storage medium connectable to the computer's system bus or via a network. Non-transitory computer-readable storage media include, for example, any type of disk, such as a magnetic disk (e.g., a floppy disk, a hard disk drive (HDD), etc.), an optical disk (e.g., a CD-ROM, a DVD disk, a Blu-ray disk), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, or any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0071] 100 Information Processing Systems 200···Network 300 Server 301···Communications Department 302...Storage section 303 Control section 400 User terminal

Claims

1. An information processing device for evaluating development of a subject user including a newborn, acquiring user biometric information, which is predetermined biometric information of the subject user, detected by a predetermined detection means; acquiring activity status information, which is information regarding the activity status of the subject user, including information regarding a predetermined state quantity and physical activity of the subject user, based on the user biological information; determining an activity state change pattern, which is a change pattern of the activity state exhibited by the subject user, based on the activity state information; assessing the subject user's development based on the activity state change pattern; a control unit that executes the The control unit acquiring a state based on the Brazelton Newborn Behavioral Rating Scale using, as the state quantity of the subject user, coordinate information of the subject user's body parts, or / and coordinate information of the subject user's face parts, or / and information on vocalizations by the subject user; As the information regarding the physical activity of the subject user, information regarding the movement state of the subject user's limbs, head, and trunk is acquired. Information processing device.

2. The control unit When acquiring the state quantity of the test user, the state is acquired based on an eye opening state of the test user based on a vertical distance between the eyes of the test user. The information processing device according to claim 1 .

3. The control unit When acquiring the state quantity of the test user, the state is acquired based on an emotional state of the test user based on the vertical and horizontal distances of the test user's mouth, or / and the vertical and horizontal distances of the test user's eyebrows, or / and the vertical and horizontal distances of the test user's eyes. The information processing device according to claim 1 .

4. The control unit When acquiring the state quantity of the test user, the state is acquired based on a frequency band of the crying voice of the test user. The information processing device according to claim 1 .

5. The control unit determining a pain onset period as the activity state change pattern based on the subject user's alertness, sedation, vocalization, body movement, facial tension, and muscle tension; evaluating the effectiveness of a medical treatment for the subject user based on the activity state change pattern; The information processing device according to claim 1 .

6. The control unit The activity state change pattern is determined by inputting the user's biometric information into a pre-learning model constructed by a neural network model having an input layer that accepts input of predetermined input image data, an intermediate layer that extracts features representing information related to the activity state of the subject user from the input image data, and an output layer that outputs a classification result based on the features. The information processing device according to claim 1 .

7. An information processing method for evaluating development of a subject user, including a newborn, comprising: The computer acquiring user biometric information, which is predetermined biometric information of the subject user, detected by a predetermined detection means; acquiring activity status information, which is information regarding the activity status of the subject user, including information regarding a predetermined state quantity and physical activity of the subject user, based on the user biological information; determining an activity state change pattern, which is a change pattern of the activity state exhibited by the subject user, based on the activity state information; assessing the subject user's development based on the activity state change pattern; Acquiring a state based on the Brazelton Newborn Behavioral Rating Scale using coordinate information of the subject user's body parts, or / and coordinate information of the subject user's face parts, or / and information on vocalizations by the subject user as the state quantity of the subject user; Acquiring information about the subject user's physical activity, including information about the motion states of the subject user's limbs, head, and trunk; An information processing method that performs the above.

8. An information processing program for evaluating the development of a test user, including a newborn, comprising: On the computer, acquiring user biometric information, which is predetermined biometric information of the subject user, detected by a predetermined detection means; acquiring activity status information, which is information regarding the activity status of the subject user, including information regarding a predetermined state quantity and physical activity of the subject user, based on the user biological information; determining an activity state change pattern, which is a change pattern of the activity state exhibited by the subject user, based on the activity state information; assessing the subject user's development based on the activity state change pattern; Acquiring a state based on the Brazelton Newborn Behavioral Rating Scale using coordinate information of the subject user's body parts, or / and coordinate information of the subject user's face parts, or / and information on vocalizations by the subject user as the state quantity of the subject user; Acquiring information about the subject user's physical activity, including information about the motion states of the subject user's limbs, head, and trunk; An information processing program that executes the above.

Images