System, method, and apparatus for forming a machine learning session

Abstract

The disclosed embodiments relate to a system, method, and apparatus for forming an ML session so that an AI system can identify, track, and diagnose a user through the user's toilet session. As used herein, a session is the period during which a user is active in the toilet. A session may include a specific inactivity if the inactivity continues for a predetermined period. In an exemplary embodiment, the system is configured to image excrement and classify toilet use sessions using digital biomarkers (DBMs) by an ML algorithm. The ability to create an excrement log to accurately provide detailed information to physicians and healthcare providers can bring about a revolution in healthcare by notifying when further urine or fecal screening is needed. The disclosed embodiments of image identification via ML and AI provide an association between medical records and specific excrement patterns without the need to change the user's habits.

Description

Technical Field

[0001] This application claims priority to U.S. Provisional Application Serial No. 63 / 339,407, filed May 6, 2022. This application is a Continuation-in-Part (CIP) of U.S. Application Serial No. 17 / 701,799, filed March 23, 2022, which claims priority to Application Serial No. 16 / 446,111 (Patent No. 11,298,112), filed June 19, 2019, which is a continuation of Application Serial No. 16 / 016,559 (Patent No. 10,376,246), filed June 23, 2018, which is a continuation of PCT / US2018 / 026618, filed April 6, 2018, which claims priority to Provisional Application Serial No. 62 / 482,912, filed April 7, 2017. This application is also a Continuation-in-Part of Application Serial No. 17 / 432,955, filed February 10, 2022, which claims priority under 35 U.S.C. § 371 to PCT Application Serial No. PCT / US2020 / 019383, which claims priority to Provisional Application Nos. 62 / 809,522 (filed February 22, 2019), 62 / 900,309 (filed September 13, 2019), and 62 / 959,139 (filed January 9, 2020). The specification of each of the foregoing applications is hereby incorporated by reference in its entirety.

Background Art

[0002] Many health conditions can be detected from the visual characteristics of human excreta. The best way to reduce costs and effectively treat these conditions is early diagnosis and treatment. Current clinical management for these and many other conditions involves monitoring and recording specific characteristics of excreta such as urine color, urination frequency, urination time, stool color, stool frequency, and stool hardness.

[0003] Recording these characteristics is currently an effective way to screen for and track conditions such as urinary tract infections, infectious diarrhea, dehydration, chronic kidney disease, cancer, gastrointestinal bleeding, inflammatory bowel disease, and constipation. Many of these conditions can lead to hospitalization, readmission, and disease progression if not diagnosed and treated early. Today, recording these characteristics is often labor-intensive and abnormalities are not identified accurately and in a timely manner. For example, it has been demonstrated that in elderly care facilities, clinically problematic changes in excrement are often not identified within 24 hours of onset.

Brief Description of the Drawings

[0004] Various embodiments of the disclosed principles will be described with reference to the following illustrative and non-limiting drawings in which like elements are assigned like reference numerals.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 6

Figure 7

Figure 8

Figure 9

[0005] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments. However, the various embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits are not described in detail so as not to obscure the specific embodiments. Further, the various aspects of the embodiments can be implemented using a variety of means, such as semiconductor integrated circuits (“hardware”), computer-readable instructions programmed in one or more programs (“software”), or some combination of hardware and software. For the purposes of the present disclosure, references to “logic” are meant to include hardware, software, firmware, or combinations thereof. References to circuits or microcircuits are intended to include the hardware and optionally software necessary to perform the desired tasks. Such circuits can also be used for wireless communication from a toilet to one or more remote servers configured to receive the acquired information and obtain the desired information. The machine language or artificial intelligence disclosed herein may be executed remotely (e.g., on a server or processor) or incorporated within the toilet as disclosed herein.

[0006] References to “one embodiment,” “an embodiment,” “exemplary embodiment,” “various embodiments,” etc., indicate that the embodiments of the invention so described may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Further, some embodiments may have some or all of the features described for other embodiments, or may not have them.

[0007] In the following description and claims, the terms "coupled" and "connected" may be used along with their derivatives. It should be understood that these terms are not intended to be synonyms of each other. Rather, in certain embodiments, "connected" is used to indicate that two or more elements are in direct physical or electrical contact with each other. "Coupled" is used to indicate that two or more elements cooperate or interact with each other, and may or may not have physical or electrical components therebetween.

[0008] When used in the claims, unless otherwise specified, the use of ordinal adjectives such as "first," "second," "third," etc. to describe common elements merely indicates that different examples of similar elements are being referred to, and does not mean that the elements so described must be in a given order in any temporal, spatial, ranking, or other manner.

[0009] Discussions herein using terms such as, for example, "processing," "computing," "calculating," "determining," "establishing," "analyzing," "checking," etc. may refer to operations and / or processes of a computer, computing platform, computing system, or other electronic computing device, where the operations and / or processes manipulate and / or transform data represented as a physical (e.g., electronic) quantity within a register and / or memory of the computer into other data similarly represented as a physical quantity within a register and / or memory of the computer, or other data similarly represented as a physical quantity within another information storage medium capable of storing instructions for executing the process.

[0010] The disclosed embodiments relate to systems, methods, and apparatuses for forming an ML session so that an AI system can identify, track, and diagnose a user through the user's toilet session (as used herein, "event(s)" or "session(s)"). As used herein, a session is the period during which a user is active in the toilet. A session may include a particular inactivity if the inactivity continues for a predetermined period.

[0011] In an exemplary embodiment, the system is configured to image excreta and classify toilet use sessions using digital biomarkers (DBMs) by an ML algorithm. The ability to create an excreta log to accurately provide detailed information to physicians and healthcare providers can bring about a revolution in healthcare by notifying when further urine or fecal screening is needed. The disclosed embodiments of image identification via ML and AI provide an association between a user's medical records and specific excreta patterns without the need to change the user's habits.

[0012] In certain embodiments, the system can determine the relevance between these excreta logs and the onset of specific diseases. In the exemplary studies reported herein, excreta records of an exemplary system were collected and compared against each other and against the anonymized medical records of patients to provide correlation data between acute onset such as cloudy urine, occult blood, diarrhea, and constipation and the excreta logs. This correlation can establish state thresholds for each type of acute onset and individual state thresholds for reporting to healthcare workers.

[0013] In an exemplary embodiment, the present disclosure uses an Internet-connected replaceable toilet seat that uses time-lapse imaging to continuously capture defecation and urination. The toilet seat may comprise one or more sensors. The sensor device may incorporate hardware (i.e., the toilet seat) as well as software components (i.e., analysis software created via a machine learning model). After the toilet sensor captures time-lapse images of the contents entering the toilet bowl, these images are transferred to the software components of the system. The software components analyze the captured images and report characteristics of feces and urine outside the normal range, such as frequency of excretion, fecal hardness, fecal color, urine color, etc.

[0014] Data captured by the toilet sensor may be transferred to a cloud server. The data results may then be further analyzed on the cloud server and then analyzed by a human reviewer to verify the identified fecal and urine characteristics. In the specific embodiments reported here, the data is analyzed on one or more cloud servers, but the disclosed principles are not limited to this. For example, the data may be collected and analyzed and stored locally on the device.

[0015] The detection system may comprise one or more light sources, lenses, optical trains, toilet seats, sensors (e.g., temperature, distance, capacitance, bioelectrical impedance), fixed brackets, analysis circuits, and communication circuits. In one embodiment, the bioelectrical impedance indicates the bioelectrical impedance passing through the user's body while the user is sitting on the toilet. For this purpose, general bioelectrical impedance measurements may be used. The circuit may comprise one or more electronic circuits (e.g., a microprocessor and a memory circuit) configured to receive an optical image from the lens and the optical train, convert the image into digital data, and optionally analyze the converted digital data according to a predetermined algorithm.

[0016] A detection system including an optical component, software / firmware, and a sensor may be integrated with the toilet seat or may be attached to the toilet seat or toilet using fixtures and brackets. In an integrated embodiment, the toilet seat may include a circuit including hardware, software, and firmware configured to collect data, analyze the data, and store the data and its results. The communication module added to the detection device may further include a communication circuit for communicating the data results intermittently or as needed, for example wirelessly.

[0017] FIG. 1 is an exemplary toilet seat according to an embodiment of the present disclosure. Specifically, FIG. 1 shows a toilet seat assembly 100 having a lid 110, a rim 112, and a detection system 120. The assembly 100 may be integrated with a toilet (not shown) or added to a toilet (not shown). The lid 110 may further include sensor components (not shown) for opening when a user is nearby. The detection system 120 may include various modules for performing the disclosed steps. In one application, the detection system 120 may include an optical detection module (not shown) including one or more lenses (as an optical train) and / or an IR detection module for detecting or visualizing urine or excrement. The detection system 120 may also include an illumination source, a temperature sensor (e.g., temperature), an analysis circuit, and a communication circuit. A detection sensor can be fixed to the rim 112 or a toilet (not shown) using a fixing bracket (not shown). The detection system 120 may also include one or more electronic circuits (e.g., a microprocessor and a memory circuit) configured to receive an optical image from an optical train (not shown) and convert the optical image into digital data. The electronic circuit(s) may be configured to analyze the converted digital data according to a predetermined algorithm according to the disclosed principle. The detection system 120 may include a memory component for storing and executing a predetermined algorithm. As will be described later, the algorithm may implement AI to identify a disease or abnormality of interest. In one embodiment, the AI algorithm is stored in a memory circuit as software or firmware (not shown) incorporated into the detection system 120. Refer to PCT / US2018 / 026618 (filed on April 6, 2018) and PCT / US2020 / 019383 (filed on February 22, 2020) of the international application, and the specifications of these are incorporated herein by reference in their entirety for background information.

[0018] In an exemplary embodiment, the toilet may include a so-called guest button 130 that can be used by a guest (an occasional user) to start a guest session. The guest button is pressed or otherwise operated to start the acquisition and storage of data from the guest user. An additional interface (not shown) may be provided so that the guest can enter data and thereby associate the guest's data with the guest's ID. Also, near-field wireless communication or Bluetooth may be used to detect guest information from the guest's electronic device (e.g., a smartphone or smartwatch). In such an embodiment, the toilet system may communicate directly with the user's smart device and request permission to identify the guest user. Thereafter, the user may optionally give the system permission to associate and identify the recorded guest data. In one embodiment, the present disclosure relates to an algorithm that can be implemented as AI to identify a disease or abnormality of interest. The AI application may be implemented as software or firmware in one or more electronic circuits to identify and detect a disease or health abnormality of a toilet user. Further, the toilet habits and excreta of the user may be used to track the user's health over time. The AI may be trained using ML in accordance with the disclosed embodiments.

[0019] Figure 2 is a data collection flowchart from a user's toilet use session. This flowchart may be used to enable an ML system to call (filter) medically important events. In one exemplary embodiment, the ML system may be trained to identify important events, label them, and generate labeled results indicating the important events. Specifically, Figure 2 shows the collection of frames as sample images when the user starts using the toilet. In a particular embodiment, the user's session is recorded when the distance sensor 111 of the toilet lid 110 (Figure 1) is activated, or when a capacitance sensor (not shown) inside the rim 112 is activated. The frames may be recorded at a predetermined rate, such as 30 to 60 frames per second. Next, the frames are grouped into sequences 220. A sequence may be considered a group of frames associated with the start and end of frame capture. Thus, a sequence may include multiple frames from when the user sits on the toilet, stands up to get something (e.g., toilet paper), and returns to the toilet. A session 230 is a group of sequences with an inactive period of less than two minutes. Finally, episodes 240 are collected from various sessions. An episode 240 consists of a group of clinically important sessions. As will be described later, this data is used together with other data, especially for training algorithms in relation to ML. Heuristic ML algorithms may be applied at each level. This enables the ML system to filter important or interesting events, label such events, and generate reports.

[0020] Figure 3 schematically shows an exemplary ML annotation and reporting process. In Figure 3, the ML system filters events of interest and labels the events before reporting the filtered and labeled events to a human reviewer. Note that while the exemplary embodiment of Figure 3 suggests a human reviewer, the reviewer may be replaced by an AI system that performs the review based on prior training.

[0021] In Figure 3, frame 308, sequence 312, and session 314 may be collected from a particular user's toilet use session. Next, each of frame 308, sequence 312, and session 314 is annotated by a human observer. Next, observation types and optional nodes are assigned to each object (frame, session, or sequence) to describe a clinical or non-clinical state. This is illustrated as node 320. Next, the nodes are annotated or combined to form a review 330. Each review is an observation result and is assigned to a frame, sequence, or session. Parameter 340 defines a group of reviews of events during a fixed time window indicating clinical importance. Finally, report 350 is a group of parameters captured at a moment. As an example, if cloudy urine is observed in a particular frame, the associated frame, sequence, and session may be labeled as indicating cloudy urine, and a report may be generated therefrom. The start or stop of the appearance of cloudy urine is indicated by parameter change 340. Parameter changes include parameter changes such as changes indicating clinical features (e.g., cloudy urine or blood in the stool) or non-clinical features (e.g., from regular urination to no urination). When a parameter change is detected, additional warnings may be generated. The report may show the aggregation of all parameter changes.

[0022] FIG. 4 is an exemplary flow diagram showing a machine learning pipeline. FIG. 4 shows that, in one embodiment, the ML pipeline is hierarchical. In FIG. 4, the frames are shown as group 402. The frames 402 are collected when the user uses the toilet. The frames are numbered from 1 to n to indicate the time sequence. In hierarchy 404, the frames are labeled by ML to indicate the detected features. Hierarchies 402 and 404 show the multi-label classification of biomarkers at the individual frame level using neural networks such as perceptrons, feed-forward networks, multi-layer perceptrons, radial basis networks, convolutional neural networks, recurrent neural networks, and long short-term memory networks.

[0023] Classification based on the output of the previous stage may be used to determine the clinical session label (traditional ML variants such as random forest outperform the neural network-based approach at this stage). In hierarchy 406, various biomarkers are added as labels to the frames. This process may be extended to identify session 408 with biomarkers. Session-level classification based on low-level biomarkers and clinical event labels is shown at 410. The data may be combined at the session level to determine clinically relevant session labels: frame level (multi-label classification of biomarkers in individual frames) and session level (classification based on frame-level output to determine clinical session labels using ML variants such as random forest).

[0024] In one application of the disclosed embodiments, a two-stage process is used to identify abnormalities in a patient of interest. This process may include steps of sessionization and machine labeling processes, as schematically shown in FIGS. 5(A) and 5(B), respectively.

[0025] Figure 5A shows an exemplary sessionization process as the first of two processes. As described above, sessionization is the process of grouping frames into sequences and sessions based on time series data, behavioral data, and personalization data. Time-based sessionization can be either static or dynamic. In static time-based sessionization, static time intervals or session timeout intervals are used. This interval may be, for example, 1 minute, 2 minutes, 5 minutes, or 10 minutes. In Figure 5A, data pipeline 500 is sessionized and sent to ML system 550 (Figure 5B) for ML labeling. In this embodiment, frame 502 is sent to sessionization processor 505. Sessionization processor 505 may comprise circuitry and software configured to receive frame 502 and define sequences and sessions based on predefined parameters. Each frame may have an associated timestamp. The timestamp may indicate the capture time of the frame relative to the clock. The passage of time between frame collections is indicated by Δ in Figure 5A. The frames are arranged into sequences (e.g., Seq. 1-4) according to their timestamps. For example, frame 2 is collected 1 minute and 50 seconds after receiving frame 1, frame 3 is collected almost immediately after frame 2, a 2 minute and 30 second passage occurs between the collection of frames 3 and 4, frames 4, 5, 6 are collected substantially immediately, and a 1 minute and 55 second passage occurs before frames 7, 8 are collected. Processor 505 forms sequences according to the timestamps. Thus, sequence 1 contains only frame 1, sequence 2 contains frames 2, 3, sequence 3 contains frames 4, 5, 6, and sequence 4 contains frames 7, 8.

[0026] Furthermore, the processor 505 may sessionize frames and sequences by comparing the Δ value to a threshold value (e.g., 2 minutes). Thus, session 1 includes sequences 1, 2, and session 2 includes sequences 4 through 8. The threshold value may be dynamic, or other dynamic session timeout thresholds may be used. A dynamic timeout interval for a session may be devised according to the time bands of actions (i.e., start, end) in the toilet use session. For example, features regarding a user's specific actions may be recognized to define the dynamic threshold. Features regarding actions may include, for example, whether a person stands up during a session or whether they fall without assistance, etc.

[0027] FIG. 5B is a continuation of FIG. 5A and shows exemplary machine labeling or scoring by using an ML system according to an embodiment of the present disclosure. Specifically, FIG. 5B shows an embodiment in which each of a frame, a sequence, and a session is labeled using ML trained to identify a particular abnormality (e.g., cloudy urine or bloody stool) in a toilet bowl. Here, both time data and personal data may be used to label the collected data (frames, sequences, sessions).

[0028] The right side of FIG. 5B shows unlabeled data as frames and sequences, the left side of FIG. 5B shows the ML labeling process, and the ML process is schematically illustrated as 550. The optical data collected from the user's toilet session arrives as frame 502. The frame is received by frame labeling processor 552. Frame labeling processor 551 may comprise circuitry and software for attaching a predetermined label to the optical data frame. At line 552, the received frames are arranged from left to right in the order of their arrival, with the oldest frame on the left (clean toilet bowl), followed by urination, cloudy urine, formed stool, and formed stool, and the latest frame on the right (non-formed stool). Line 554 shows the label that matches the optical characteristics of each frame. In one embodiment, the label is selected by frame labeling processor 551. For example, frame labeling processor 551 may represent a clean toilet bowl from the corresponding frame. At line 556, each frame is labeled with the corresponding label, forming a labeled frame. The labeled frame is schematically illustrated as line 556. This step may also be performed by frame labeling processor 551.

[0029] The labeled frames 556 are grouped into segments 558. The grouping may be performed based on static or dynamic thresholds, as described with respect to FIG. 5A. One or more processors may be used to perform the grouping. The formed sequence may be combined with the selected label, as shown in row 560. In an exemplary embodiment, the solid and non-solid stools are combined into one sequence. The formed sequence 558 may be sent to a sequence labeling processor 562 to combine the sequence with a sequence label 564 to form a labeled sequence 566. It should be noted that the labeled sequence 566 includes one or more labeled frames (LFs), as shown in FIG. 5B. The combination of the labeled frames and the labeled sequence allows a healthcare provider or AI to easily collect the differentiated frames and sessions in which an abnormality is indicated in the user's stool. It should also be noted that the sequences containing these frames are labeled as stools, even though frames 4, 5, and 6 contain different physical characteristics (i.e., solid and non-solid stools). A healthcare provider (or AI) reviewing the labeled sequence 3 may be alerted that the physical characteristics of the stool have changed within a few frames.

[0030] Figure 5C shows an application of the disclosed principles for forming ML labeling to label a user's session. Here, the machine labeling processor 570 receives sessions 1 through n as input. These sessions are also illustrated as a clean toilet, cloudy urine, urination, solid feces, and a clean toilet in row 572. Although a label for a clean toilet has been added in Figure 5C, one of ordinary skill in the art will readily understand to apply a label for a clean toilet to the frames and sequences of Figures 5A and 5B, respectively. Session 572 is combined with session label 574. The machine labeling processor 570 may provide additional labels as described above and be combined with session label 574. Row 576 shows the labeled sessions identified as sessions 1, 2, 3. Each session may have one or more sequences and one or more frames. The labeled sessions, labeled sequences, and labeled frames are stored in a memory circuit and associated with the user. For example, a user's labeled sessions over several days may be used to track the user's health or to identify health abnormalities.

[0031] As discussed in connection with Figure 4, sessioning may be either static or dynamic. In static sessioning, the ML system determines the start and stop of a session according to predetermined thresholds. For example, if the threshold is set to 2 minutes (e.g., a timeout), if there is no user activity for more than 2 minutes, the session is considered to have ended.

[0032] FIG. 6 schematically shows dynamic sessionization according to an embodiment of the present disclosure. Dynamic sessionization may be considered as a variable time window that varies according to external variables including specific characteristics of a user. In one embodiment, dynamic sessionization may be considered as a series of consecutive toilet use actions of a user over a predetermined time interval. In an exemplary embodiment, the durations of the start and end of a toilet use session may be dynamically changed based on external characteristics including the user's behavior or health particularities (for example, a user with a physical disability may require a longer start duration than a healthy adult or child).

[0033] In FIG. 6, as shown by dynamic start session 602, the start of the toilet use session may be set at 3 minutes. The start of the toilet use session may be triggered by a sensor that detects the presence of a user who is in or near the toilet. This sensor may be configured to change the start of the toilet use session according to the user and the user's specific needs. Also, the actual toilet use time shown as 604 may be static or dynamic. In FIG. 6, the toilet use time 604 is shown as a static time interval (Δ) of 2 minutes, but this may be changed according to specific predefined parameters. The end 606 of the toilet use session may define a dynamic period. As explained, this part may be extended or shortened according to the user's needs or capabilities. Thus, although the exemplary representation of FIG. 6 shows session 1 as having a total of 8 minutes, this may be shortened or extended according to the user's needs.

[0034] FIG. 7 shows an exemplary embodiment of the present disclosure for providing information (e.g., training) to a machine learning system. In FIG. 7, various sensors related to the toilet provide sensor data 704 to the ML system 700. The sensor data may be obtained from sensors that collect time of flight (TOF) information 706, capacitance information 708, and information 710 from an optical train. The TOF may include information related to the length of an active session. The capacitance sensor may relay information provided by a sensor disposed on the toilet seat that activates when a user is sitting on the toilet. The optical train information may include a frame provided by an optical train installed in the toilet (see FIG. 1). An exemplary TOF sensor may detect whether a user is standing in front of the toilet, sitting on the toilet, or someone is sitting on the toilet riser. In some embodiments, the TOF sensor may be disposed on the lid of the toilet 111. For a particular toilet having a toilet riser (e.g., for an individual who may not be able to squat over the toilet), the TOF sensor may be disposed (e.g., near the toilet) to detect the presence of a user on the toilet riser.

[0035] The ML system 700 also receives annotations from an observer. The annotations may include frames labeled by a human observer (e.g., blood in stool, cloudy urine, clean toilet bowl, etc.). The output of the ML system 700 is a label, shown as labels for frames, sequences, and sessions. In the exemplary embodiment of FIG. 7, four label categories of base 720, action 722, system 724, and clinical 726 are shown. Other labels may be added without departing from the disclosed principles. Labels may be added to each frame, sequence, and session as needed to provide a comprehensive understanding of the user's toilet usage behavior and assist in potential diagnosis.

[0036] The base session 720 may include labels for base observations during a toilet use session. For example, the base session may include formed stool, unformed stool, presence of urine, presence of urination, etc. The action label 722 may include labels for the user's actions during a toilet use session. This group may include labels including standing urination, sitting urination, non - water wash, water wash, etc. The system label 724 relates to the state of the system during a toilet use session. The system session may include labels including a clean toilet bowl, a clean lens, a dirty toilet bowl, a dirty lens, etc. The clinical label 726 includes labels for clinical associations during a toilet use session and may include, for example, scatole, presence of blood, cloudy urine, etc. Referring now to FIGS. 5A - 5C and FIG. 7, the labels may be added by each of the ML processors 552, 562, and 270 in addition to the various labels schematically shown in FIG. 7.

[0037] As an additional example, the actions of a user during toilet use mainly due to how to wipe with toilet paper can give rise to multiple sequences. Using multiple sensors and time - based sessioning, this user behavior can be correctly sessioned. This may be of particular interest to the elderly population who may require assistance when wiping and washing. Behaviors such as delayed or forgotten water wash can be sessioned using the ML system and sequence labels. Similarly, not performing a water wash is a user behavior that causes inaccurate sessioning. To solve this problem, data from the previous session may be input into the analysis of the current session to identify only the new excreta of the new session. In certain embodiments, sessioning may utilize multiple sensor data sets to reduce other user actions such as walking in front of the toilet or drying after a shower (if the shower is near the toilet).

[0038] FIG. 8 shows exemplary session rules for taking into account user actions. Specifically, FIG. 8 shows various examples in which user actions (or lack thereof) or water washing can affect the labeling process. Without departing from the disclosed principles, additional rules for labeling or processing frame information prior to labeling may be added.

[0039] FIG. 9 schematically shows an exemplary implementation of the personalization rules. The left side of FIG. 9 shows inputs that can be used as indicators for user identification. The inputs may include personal attributes of the user. One or more attributes may be used to identify the user. Exemplary indicators in FIG. 9 include weight (902), height (904), non-water washing (805), and bioelectrical impedance (908). Other exemplary indicators that can be known from the user's profile include age, excretion status, current medication, mobility, body temperature (measured by a toilet seat or thermal scanner), or other available health information. In an exemplary embodiment, the urine output of the user may be measured and used as an indicator. In yet another embodiment, the result of a urine dipstick (or other litmus) test of the user may be used as an indicator. In one embodiment of the present disclosure, watery stools or persistent watery stools may be identified by measuring the bioelectrical impedance of substances in the toilet. Another user indicator may be the user's voice (not shown) or fingerprint (not shown). The user may start a session by personalizing the session. Further, the sessionization may be personalized for users who do not perform proper flushing (910). Similar user recognition may also be used for users who need to sit on the toilet to dry off after taking a shower. The sessionization may be based on the user's toilet usage activities. For example, users who regularly use the toilet in the morning or evening may be identified based on the usage time. The use of personal attributes to identify users may be important when the toilet is shared in a facility (e.g., a nursing home). The right side of FIG. 9 relates to actions performed by the sessionization processor, and the labeling actions include, as an example, a non-flushing label (910), a user action label (912), and other exemplary labels (914).

[0040] When frames, sequences, and sessions are annotated, a report regarding the user's toilet usage session may be generated. Depending on the desired report parameters, the report may include the latest session or multiple sessions. In one embodiment, if certain parameters are observed in a session, the report may trigger a warning. For example, if the user's session shows significant blood in the stool, medical personnel may be notified.

[0041] The annotated report may be generated by learning the user's session. The filtering session may generate a frame of interest for the annotation system. The annotation system may collect relevant frames of interest for the report. Errors may be detected and corrected before generating the report. For example, if the frame identifies a health anomaly (e.g., bloody stool) but the associated sequence and session labels suggest a dirty lens, ML filtering may explain the anomaly while considering other physical problems. Finally, the generated sequence may be searched to easily examine the state shown by the patient.

[0042] The following examples are provided to further illustrate the application of the disclosed principles to various exemplary embodiments.

[0043] Example 1 relates to a method of automatically labeling a user's toilet usage behavior, including activating sensors to record the toilet usage behavior in multiple frames, each frame capturing one or more activities during a discrete time interval, each frame being separated from subsequent frames by a time interval, forming one or more sequences by grouping the frames separated from each other at substantially constant time intervals, and forming one or more sessions by grouping the sequences according to specific intervals, labeling each frame by identifying one or more states present within the frame, labeling each sequence by identifying one or more states present among the multiple frames within each sequence, and labeling each session by identifying one or more states present among the multiple sequences within each session.

[0044] Example 2 relates to the method of Example 1, and further includes activating a sensor to record the toilet usage behavior, which includes activating a capacitance sensor to generate the start of a toilet usage session, and activating a time-of-flight sensor in conjunction with an optical system for recording the toilet usage session.

[0045] Example 3 relates to the method of Example 1, and further includes filtering the plurality of frames, sequences, and sessions according to a predetermined state to generate a state report.

[0046] Example 4 relates to the method of Example 3, and further includes sending the state report to a provider when the state exceeds a threshold.

[0047] Example 5 relates to the method of Example 1, and the one or more states are selected to identify biomarkers or state markers.

[0048] Example 6 relates to the method of Example 5, and the state marker is selected from the group consisting of bloody stool, non-formed stool, cloudy urine, and hematuria.

[0049] Example 7 relates to the method of Example 1, and the state includes physical characteristics of excrement.

[0050] Example 8 relates to the method of Example 1, and the threshold is dynamically determined according to one or more user characteristics.

[0051] Example 9 relates to the method of Example 8, and the one or more characteristics include age, weight, and mobility.

[0052] Example 10 relates to the method of Example 1, and the sensor captures the user's activities in the toilet using an optical train and a circuit for receiving and storing image data.

[0053] Example 11 relates to the method of Example 1, and the frame includes an optical image of one or more of the user's excrement or urine.

[0054] Example 12 relates to a system for automatically labeling a user's toilet usage behavior, comprising a processor circuit and a memory circuit communicating with the processor circuit, the memory circuit being configured to have instructions for causing the processor circuit to automatically label the user's toilet usage behavior by the following operations, and operating a sensor to record the toilet usage behavior in a plurality of frames, each frame capturing one or more activities during a discrete time interval, each frame being separated from subsequent frames by a time interval, forming one or more sequences by grouping the frames separated from each other at substantially constant time intervals, forming one or more sessions by grouping the sequences according to specific intervals, labeling each frame by identifying one or more states present in the frame, labeling each sequence by identifying one or more states present in the plurality of frames within each respective sequence, and labeling each session by identifying one or more states present in the plurality of sequences within each respective session.

[0055] Example 13 is directed to the system of Example 12, and the memory circuit further comprises instructions for operating the sensor to record the toilet usage behavior by operating a capacitance sensor to generate the start of a toilet usage session and operating a time-of-flight sensor in conjunction with an optical system to record the toilet usage session.

[0056] Example 14 is directed to the system of Example 12, and further includes filtering the plurality of frames, sequences and sessions according to a predetermined state to generate a state report.

[0057] Example 15 is directed to the system of Example 14, and further includes sending the state report to a provider when the state exceeds a threshold.

[0058] Example 16 is directed to the system of Example 12, and the one or more states are selected to identify a biomarker or a state marker.

[0059] Example 17 is directed to the system of Example 16, and the state marker is selected from the group consisting of bloody stool, non-formed stool, cloudy urine, and hematuria.

[0060] Example 18 is directed to the system of Example 12, and the state includes physical characteristics of excrement.

[0061] Example 19 is directed to the system of Example 12, and the threshold is determined dynamically according to one or more user characteristics.

[0062] Example 20 is directed to the system of Example 19, and the one or more characteristics are selected from the group consisting of the user's age, excretion status, current medication, mobility, body temperature, or other available health information.

[0063] Example 21 is directed to the system of Example 12, and the sensor captures the user's activities in the toilet using an optical train and a circuit for receiving and storing image data.

[0064] Example 22 is directed to the system of Example 12, and the frame includes an optical image of one or more of the user's excrement or urine.

[0065] Example 23 is directed to a method of forming a machine learning (ML) session from user activities, including receiving a plurality of frames from a sensor, each frame capturing one or more activities during a discrete time interval, each frame being separated from subsequent frames by a time interval, grouping the frames separated from each other at substantially constant time intervals to form one or more sequences, and grouping the sequences according to specific intervals exceeding a threshold to form one or more sessions.

[0066] Example 24 is directed to the method of Example 23, and the threshold value is static and pre-determined.

[0067] Example 25 is directed to the method of Example 23, and the threshold value is dynamic.

[0068] Example 26 is directed to the method of Example 25, and the threshold value is dynamically determined according to one or more user characteristics.

[0069] Example 27 is directed to the method of Example 26, and the one or more characteristics are selected from the group consisting of the user's age, excretion status, current medication, mobility, body temperature, or other available health information.

[0070] Example 28 is directed to the method of Example 23, and the frame is collected by one or more sensors for capturing the user's activities in the toilet.

[0071] Example 29 is directed to the method of Example 23, and the frame includes optical images of one or more of the user's excreta or urine.

[0072] Example 30 is directed to a system for forming a machine learning (ML) session from user activities, comprising a processor circuit and a memory circuit communicating with the processor circuit, the memory circuit being configured to have instructions for causing the processor circuit to perform the following operations: receiving a plurality of frames from sensors, each frame capturing one or more activities during discrete time intervals, each frame being separated from subsequent frames by a substantially constant time interval, forming one or more sequences by grouping the frames separated from each other by a specific interval, and forming one or more sessions by grouping the sequences according to specific intervals exceeding a threshold value.

[0073] Example 31 is directed to the system of Example 30, and the threshold value is static and pre-determined.

[0074] Example 32 is directed to the system of Example 30, and the threshold value is dynamic.

[0075] Example 33 is directed to the system of Example 32, and the threshold value is dynamically determined according to one or more user characteristics.

[0076] Example 34 is directed to the system of Example 33, and the one or more characteristics are selected from the group consisting of the user's age, excretion status, current medication, mobility, body temperature, or other available health information.

[0077] Example 35 is directed to the system of Example 30, and the frame is collected by one or more sensors for capturing the user's activities in the toilet.

[0078] Example 36 is directed to the system of Example 30, and the frame includes optical images of one or more of the user's excreta or urine.

[0079] Example 37 is directed to a non-transitory computer-readable storage medium including a processor circuit and a memory circuit that communicates with the processor circuit and includes instructions for forming a machine learning (ML) session from user activities. The memory circuit further includes instructions for causing the processor to perform the following operations: receiving a plurality of frames from sensors, each frame capturing one or more activities during discrete time intervals, each frame being separated from subsequent frames by a substantially constant time interval, forming one or more sequences by grouping the frames separated from each other by exceptional intervals, and forming one or more sessions by grouping the sequences according to exceptional intervals exceeding a threshold value.

[0080] Example 38 is directed to the medium of Example 37, and the threshold value is static and pre-determined.

[0081] Example 39 is directed to the medium of Example 37, and the threshold value is dynamic.

[0082] Example 40 is directed to the medium of Example 39, and the threshold value is dynamically determined according to one or more user characteristics.

[0083] Example 41 is directed to the medium of Example 40, and the one or more characteristics are selected from the group consisting of the user's age, excretion status, current medication, mobility, body temperature, or other available health information.

[0084] Example 42 is directed to the medium of Example 37, and the frame is collected by one or more sensors for capturing the user's activities in the toilet.

[0085] Example 43 is directed to the medium of Example 37, and the frame includes optical images of one or more of the user's excrement or urine.

Claims

1. 1. A method for automatically labeling a user's toilet usage activity, comprising: activating a sensor to record the toilet use activity in a plurality of frames, each of the frames capturing one or more activities during a discrete time interval, each of the frames separated from a subsequent frame by a time interval; forming one or more sequences by grouping a plurality of said frames separated from one another by substantially regular time intervals; and forming one or more sessions by grouping the sequences according to distinct intervals; labeling each of said frames by identifying one or more states present in said frame; labeling each of the sequences by identifying one or more states present in a plurality of the frames within each of the sequences; labeling each of said sessions by identifying one or more states present in one or more of said sequences within each said session.

2. 10. The method of claim 1, wherein activating a sensor to record the toilet usage activity further comprises activating a capacitance sensor to generate a start of a toilet usage session, and activating a time-of-flight sensor in conjunction with an optical system to record the toilet usage session.

3. The method of claim 1 , further comprising filtering the frames, the sequence(s), and the session(s) according to a predetermined condition to generate a condition report.

4. The method of claim 3 , further comprising sending the status report to a provider when each of the conditions exceeds a threshold.

5. The method of claim 1, wherein one or more of the conditions are selected to identify biomarkers or condition markers.

6. 6. The method of claim 5, wherein the condition marker is selected from the group consisting of bloody stool, unformed stool, cloudy urine, and hematuria.

7. The method of claim 1 , wherein each of the conditions comprises a physical characteristic of the waste material.

8. The method of claim 4 , wherein the threshold is dynamically determined as a function of one or more user characteristics.

9. The method of claim 8, wherein one or more of the user characteristics include age, weight, and mobility.

10. The method of claim 1 , wherein the sensor captures the user's activities in the restroom using an optical train and circuitry that receives and stores image data.

11. The method of claim 1 , wherein each of the frames includes an optical image of one or more of the user's waste or urine.

12. 1. A system for automatically labeling a user's toilet usage activity, comprising: a processor circuit; a memory circuit in communication with the processor circuit, the memory circuit configured with instructions to cause the processor circuit to automatically label the toilet usage activity of the user by: activating a sensor to record the toilet use activity in a plurality of frames, each of the frames capturing one or more activities during a discrete time interval, each of the frames separated from a subsequent frame by a time interval; forming one or more sequences by grouping a plurality of said frames separated from one another by substantially regular time intervals; grouping the sequences according to distinct intervals to form one or more sessions; labeling each frame by identifying one or more states present in said frame; labeling each of the sequences by identifying one or more states present in a plurality of the frames within each of the sequences; The system labels each of the sessions by identifying one or more states present in one or more of the sequences within each of the sessions.

13. 13. The system of claim 12, wherein the memory circuit further comprises instructions for activating a capacitance sensor to generate a start of a toilet use session and a time-of-flight sensor in conjunction with an optical system to record the toilet use activity by activating the sensors to record the toilet use session.

14. The system of claim 12 , further comprising filtering a plurality of the frames, one or more of the sequences, and one or more of the sessions according to a predetermined condition to generate a condition report.

15. The system of claim 14 , further comprising sending the status report to a provider when each of the conditions exceeds a threshold.

16. The system of claim 12, wherein one or more of the conditions are selected to identify a biomarker or condition marker.

17. 17. The system of claim 16, wherein the condition markers are selected from the group consisting of bloody stool, unformed stool, cloudy urine, and hematuria.

18. The system of claim 12 , wherein each of the conditions includes a physical characteristic of the waste material.

19. The system of claim 15 , wherein the threshold is dynamically determined as a function of one or more user characteristics.

20. The system of claim 19, wherein one or more of the user characteristics are selected from the group consisting of the user's age, bowel status, current medication, mobility, body temperature, or other available health information.

21. 13. The system of claim 12, wherein the sensor captures the user's activities in the restroom using an optical train and circuitry that receives and stores image data.

22. The system of claim 12 , wherein each of the frames includes an optical image of one or more of the user's waste or urine.

23. 1. A method for forming a machine learning (ML) session from user activity, comprising: receiving a plurality of frames from a sensor, each of the frames capturing one or more activities during a discrete time interval, each of the frames separated from a subsequent frame by a time interval; forming one or more sequences by grouping a plurality of said frames separated from one another by substantially regular time intervals; and forming one or more of the machine learning (ML) sessions by grouping the sequences according to distinct intervals that exceed a threshold.

24. The method of claim 23 , wherein the threshold is static and predetermined.

25. 24. The method of claim 23, wherein the threshold is dynamic.

26. The method of claim 25 , wherein the threshold is dynamically determined as a function of one or more user characteristics.

27. ​​The method of claim 26, wherein one or more of the user characteristics are selected from the group consisting of the user's age, bowel status, current medication, mobility, body temperature, or other available health information.

28. The method described in claim 23, wherein the plurality of frames are collected by one or more sensors for capturing the user's activity in the toilet.

29. 24. The method of claim 23, wherein each of the frames includes an optical image of one or more of the user's waste or urine.

30. 1. A system for forming a machine learning (ML) session from user activity, comprising: a processor circuit; a memory circuit in communication with the processor circuit, the memory circuit configured with instructions for causing the processor circuit to: receiving a plurality of frames from the sensor, each of the frames capturing one or more activities during a discrete time interval, each of the frames separated from a subsequent frame by a substantially constant time interval; forming one or more sequences by grouping a plurality of said frames separated from one another by distinct intervals; The system forms one or more of the machine learning (ML) sessions by grouping the sequences according to distinct intervals that exceed a threshold.

31. 31. The system of claim 30, wherein the threshold is static and predetermined.

32. 31. The system of claim 30, wherein the threshold is dynamic.

33. 33. The system of claim 32, wherein the threshold is dynamically determined as a function of one or more user characteristics.

34. The system described in claim 33, wherein one or more of the user characteristics are selected from the group consisting of the user's age, bowel status, current medication, mobility, body temperature, or other available health information.

35. The system described in claim 30, wherein the plurality of frames are collected by one or more sensors for capturing the user's activity in the toilet.

36. 31. The system of claim 30, wherein each of the frames includes an optical image of one or more of the user's waste or urine.

37. 1. A non-transitory computer-readable storage medium comprising: a processor circuit; and a memory circuit in communication with the processor circuit and comprising instructions for forming a machine learning (ML) session from user activity, the memory circuit further comprising instructions for causing the processor circuit to: receiving a plurality of frames from the sensor, each of the frames capturing one or more activities during a discrete time interval, each of the frames separated from a subsequent frame by a substantially constant time interval; forming one or more sequences by grouping said frames separated from one another by distinct intervals; The medium forms one or more of the machine learning (ML) sessions by grouping the sequences according to distinct intervals that exceed a threshold.

38. 38. The medium of claim 37, wherein the threshold is static and predetermined.

39. 38. The medium of claim 37, wherein the threshold is dynamic.

40. 40. The medium of claim 39, wherein the threshold is dynamically determined as a function of one or more user characteristics.

41. The medium described in claim 40, wherein one or more of the user characteristics are selected from the group consisting of the user's age, bowel status, current medication, mobility, body temperature, or other available health information.

42. The medium described in claim 37, wherein the plurality of frames are collected by one or more sensors for capturing the user's activity in the toilet.

43. 38. The medium of claim 37, wherein each of the frames includes an optical image of one or more of the user's waste or urine.

Images