Pose detection using thermal data

Abstract

The system may provide functionality for posture detection by using a posture detection algorithm on the reduced resolution images. The method may include receiving an image of a human from a sensor, receiving a bounding box placement on the image, the bounding box containing pixel data of the human in the image, obtaining bounding box data from within the bounding box, and determining a posture of the human based on the bounding box data.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to and the benefit of U.S. Ser. No. 17 / 708,493, filed March 30, 2022 and entitled "POSE DETECTION USING THERMAL DATA." U.S. Ser. No. 17 / 708,493 is a continuation-in-part of, and claims the benefit of, U.S. Ser. No. 17 / 516,954, filed November 2, 2021 and entitled "USER INTERFACE FOR DETERMINING LOCATION, TRAJECTORY AND BEHAVIOR." U.S. Ser. No. 17 / 516,954 is a continuation-in-part of, and claims the benefit of, U.S. Ser. No. 17 / 232,551, filed April 16, 2021 and entitled "THERMAL DATA ANALYSIS FOR DETERMINING LOCATION, TRAJECTORY AND BEHAVIOR." U.S. Ser. No. 17 / 232,551 is a continuation of U.S. Ser. No. 17 / 178,784, filed February 18, 2021, and entitled "MONITORING HUMAN LOCATION, TRAJECTORY AND BEHAVIOR USING THERMAL DATA" (also known as U.S. Pat. No. 11,022,495, issued June 1, 2021). U.S. Ser. No. 17 / 178,784 claims priority to U.S. Provisional Ser. No. 62 / 986,442, filed March 6, 2020, and entitled "MULTI-WIRELESS-SENSOR SYSTEM, DEVICE, AND METHOD FOR MONITORING HUMAN LOCATION AND BEHAVIOR," all of which are incorporated herein by reference in their entirety for all purposes.

[0002] The present disclosure relates generally to detecting posture using thermal data, and more specifically, to using posture information for fall detection and other activity analysis. [Background technology]

[0003] Some businesses may try to use basic machines to count the number of people entering and exiting a particular door leading to a store, but such information is very limited to analyze the actions of those people within the store. Businesses may be very interested in understanding customer movements, trajectories, and activities within their stores more deeply. For example, a business may be interested in knowing if a display in a particular aisle within a store attracted more customers to that aisle. Also, a business may be interested in knowing how many customers walked through aisle #4 and also through aisle #5, and how many customers walked through aisle #4, passed aisle #5, and then walked through aisle #6 instead. Such data may help a business optimize its operations and maximize its profits.

[0004] Businesses may also be interested in understanding general traffic patterns in their stores more completely over time. To help an entity better allocate its own resources and optimize its business relationships with collaborating third parties, an entity may want to understand traffic patterns during the busiest hours throughout the day, during the busiest days of the week, during a particular month, and / or during a particular year. Also, to help recognize anomalous behavior and / or detect incidents in real time, an entity may want to have more information about spatial and / or temporal patterns of traffic and occupancy levels.

[0005] Additionally, to analyze the health status of residents and determine whether they are certified to live independently, senior care housing providers often desire to obtain spatial and temporal movement data of tenants. For example, the provider may desire to analyze the tenant's movement speed based on the tenant's indoor location at any given time, calculate the total calories consumed based on the tenant's movement, and / or monitor the tenant's temperature.

[0006] Detecting human posture can also be useful for fall detection and other activity analysis. However, posture detection is often required in a private home or other private environment where privacy is a concern. In that regard, users typically prefer that any data collection or scanning device not obtain or store personally identifiable information. Thus, non-intrusive techniques are needed to implement such fall detection analysis at home. In that regard, high-resolution cameras or other techniques may not be desirable because such cameras may use key points to obtain and / or store face data or other personally identifiable information. Key points may include a subset of points on the human skeleton that would require obtaining detailed information about the human. Identifying key points in low-resolution images is very difficult, and often impossible, and therefore high-resolution techniques are typically required to identify key points. In contrast, low-resolution data may be preferred in a home environment. An example of low-resolution data may be thermal data. Thus, there is a need to use thermal data in conjunction with algorithms that can process the thermal data to provide posture detection.

[0007] In addition to cameras, other solutions such as wristwatches (e.g., using accelerometers and gyroscopes), radar, or lidar have also been used for fall detection. However, wristwatch technology requires a person to actively wear the technology. Also, the use of radar can result in false positives (e.g., radar can be triggered by a pet). Also, radar technology is often not effective at distinguishing motionless people as part of the detection process. Furthermore, existing systems may need to analyze densely aggregated data points, which often results in low accuracy. Analysis of such dense points may be more expensive because additional computing power may be required to differentiate points in the dense point cloud. Summary of the Invention [Means for solving the problem]

[0008] In various embodiments, the system may implement a method that includes receiving, by a processor, an image of a human from a sensor; receiving, by the processor, a placement of a bounding box on the image, the bounding box containing pixel data of the human in the image; obtaining, by the processor, bounding box data from within the bounding box; and determining, by the processor, a pose of the human based on the bounding box data.

[0009] In various embodiments, the method may also include training, by the processor, a neural network to predict placement of a bounding box on the image. The method may also include training, by the processor, the neural network using the pixel data, the human thermal data, and the environmental data. The method may also include adjusting, by the processor, an algorithm of the neural network based on the environmental data, the environmental data comprising at least one of an environmental temperature, an indoor temperature, a floor plan, a non-human thermal object, a human gender, a human age, a sensor height, a human clothing, or a human weight.

[0010] In various embodiments, the sensor may obtain thermal data about the human. The user may indicate placement of a bounding box on the image. Determining the pose may relate to frames of images captured by the sensor. Determining the pose may further include determining an aggregate pose over a period of time across multiple frames. The image may be a portion of video footage of the human. Obtaining the bounding box data may include obtaining the bounding box data at least one of over time or during an initial calibration session. The pose may include at least one of sitting, standing, lying down, exercising, dancing, running, or eating.

[0011] In various embodiments, the method may further include determining, by the processor, a fall based on aggregate posture changes from at least one of a standing posture or a sitting posture to a lying posture, and the lying posture persisting for an amount of time. The method may include extracting, by the processor, differentiating features from the images in the multiple frames using pattern recognition. The method may include limiting, by the processor, image resolution based on at least one of privacy concerns, sensor power consumption, pixel data cost, bandwidth for pixel data, computation cost, or computation bandwidth. The method may include labeling, by the processor, the posture of the person in the image.

[0012] In various embodiments, the method may further include determining, by the processor, a temperature of the human in the space based on infrared (IR) energy data of IR energy from the human, determining, by the processor, location coordinates of the human in the space, comparing, by the sensor system, the location coordinates of the human with location coordinates of the fixed object, and determining, by the sensor system, that the human is a person in response to the temperature of the object being within a range and in response to the location coordinates of the human being being distinct from the location coordinates of the fixed object. The method may include analyzing, by the processor, differentiating features of a pattern from an overhead heat signature of the human to determine a posture of the human. The method may include determining, by the processor, a trajectory of the human based on changes in temperature in pixel data, the temperature being projected onto a grid of pixels. [Brief description of the drawings]

[0013] The subject matter of the present disclosure is particularly pointed out and separately claimed in the concluding portion of this specification, however, a more complete understanding of the present disclosure may best be obtained by reference to the detailed description and claims when considered in conjunction with the drawing figures.

[0014] [Figure 1A]FIG. 1A is an exemplary schematic diagram of the main components of a sensor node, which is part of an overall system, according to various embodiments.

[0015] [Figure 1B] FIG. 1B is an exemplary schematic diagram of a gateway and a microprocessor that is part of an overall system, according to various embodiments.

[0016] [Diagram 2] FIG. 2 is an exemplary data flow diagram according to various embodiments.

[0017] [Diagram 3] FIG. 3 is an exemplary system architecture, according to various embodiments.

[0018] [Figure 4A] 4A and 4B are exemplary user interfaces according to various embodiments. [Figure 4B] 4A and 4B are exemplary user interfaces according to various embodiments.

[0019] [Diagram 5] FIG. 5 is an example building layout, according to various embodiments.

[0020] [Figure 6] FIG. 6 is an example building layout showing certain sensor nodes and areas of coverage for each sensor node, according to various embodiments.

[0021] [Figure 7] FIG. 7 is an exemplary user interface illustrating an application's response to detection of a sensor node in a physical space after a user has successfully logged into the sensor node, according to various embodiments.

[0022] [Figure 8A]8A and 8B are exemplary user interfaces that prompt users to upload to the application any tangible files that may act as complementary visual information to their representation of a physical space, according to various embodiments. [Figure 8B] 8A and 8B are exemplary user interfaces that prompt users to upload to the application any tangible files that may act as complementary visual information to their representation of a physical space, according to various embodiments.

[0023] [Figure 9] FIG. 9 is an exemplary user interface showing certain sensor nodes located in a space that may be tagged with names so that users understand the context information of each sensor node in their uniform space, according to various embodiments.

[0024] [Figure 10] FIG. 10 is an exemplary user interface illustrating the ability to configure and calibrate a sensor node, according to various embodiments.

[0025] [Figure 11A] 11A-11C show example thermal signature patterns, including standing, sitting, and lying positions, according to various embodiments. [Figure 11B] 11A-11C show example thermal signature patterns, including standing, sitting, and lying positions, according to various embodiments. [Figure 11C] 11A-11C show example thermal signature patterns, including standing, sitting, and lying positions, according to various embodiments.

[0026] [Figure 12] FIG. 12 illustrates an exemplary pose inference process, according to various embodiments.

[0027] [Figure 13]FIG. 13 illustrates an exemplary fall detection process, according to various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Detailed Description In various embodiments, the system is configured to locate, track, and / or analyze the activity of living beings within an environment. The system does not require the input of personal biometric data. Although the disclosure may discuss human activity, the disclosure contemplates tracking any item that may provide infrared (IR) energy, such as, for example, an animal or any object. Although the disclosure may discuss indoor environments, the system may also track in outdoor environments (e.g., outdoor concert venues, outdoor amusement parks, etc.) or a mixture of outdoor and indoor environments.

[0029] 1A, 1B, and 3, in various embodiments, the system may include a plurality of sensor nodes 102, a gateway 135, a microprocessor 140, a computing module 350 (e.g., a cloud computing module), a database 360, and / or a user interface 370 (e.g., FIGS. 4A, 4B, 7, 8, 9, and 10). Each sensor node 102 may include an enclosure 105, an antenna 110, a sensor module 115, a switch 120, a light emitting diode (LED) 125, and a power source 130.

[0030] In various embodiments, the sensor module 115 may be any type of sensor, such as a thermopile sensor module. The thermopile sensor module 115 may include, for example, a Heimann GmbH sensor module or a Panasonic AMG8833. Each sensor module 115 may be housed within the enclosure 105. The sensor module 115 is configured to measure temperature from a distance by detecting IR energy from an object (e.g., a living organism). If the living organism has a higher temperature, the living organism will emit more IR energy. The thermopile sensing element within the thermopile sensor module 115 may include a thermocouple on a silicon chip. The thermocouple absorbs the IR energy and produces an output signal indicative of the amount of IR energy. Thus, a higher temperature causes more IR energy to be absorbed by the thermocouple, resulting in a higher signal output.

[0031] In various embodiments, the sensor node 102 interface is wireless, which may help reduce labor and material costs associated with deployment. In various embodiments, each sensor node 102 may obtain power from any power source 130. The power source 130 may power one or more sensor nodes 102. Each sensor node 102 may be separately battery powered. The batteries may be low enough powered to operate for more than two years with a single battery (e.g., a 19 kilowatt-hour battery). The battery 130 may include batteries from any manufacturer and / or PKCELL batteries, D-cell batteries, or any other battery type. The battery 130 may be contained within a battery holder (e.g., a Bulgin battery holder). The system may also measure the battery voltage of the battery 130 (e.g., a D-cell battery). The battery voltage may be measured using an analog-to-digital converter located on-board with the antenna 110 (e.g., a Midatronics Dusty PCB antenna). The system may also add a timestamp to the battery voltage data when the battery voltage measurement is obtained.

[0032] The system may be scalable to a larger footprint by adding more sensor nodes 102 to the sensor node 102 array. In various embodiments, the sensor nodes 102 may be dynamically added, and an exemplary user interface for adding sensors is described in FIG. 7. In particular, a user may add or remove a sensor node 102 to or from an established network of sensor nodes 102 at any time or any location. The sensor node 102 may be located at any location as long as the location of the sensor node 102 is within the bandwidth of the gateway 135. Thus, a larger number of sensor nodes 102 may form a mesh network and communicate with the gateway 135. The sensor nodes 102 may communicate with the gateway 135 at approximately the same time. Each new sensor node 102 that is connected to the gateway 135 may further expand the boundary of the mesh network, improve system stability, and improve system performance. Thus, a larger number of sensor nodes 102 may form a mesh network and communicate with the gateway 135. The sensor nodes 102 may communicate with the gateway 135 at approximately the same time. Each new sensor node 102 connected to the gateway 135 further expands the boundary of the mesh network, improving system stability and improving system performance. The sensor nodes 102 may be mounted or disposed on any part of a building or any object. For example, the sensor nodes 102 may be disposed in the ceiling, sidewall, or floor of a desired space using any fasteners known in the art. The distance between the sensor nodes 102 may be, for example, 4 meters apart for a ceiling that is 2.5 meters high.

[0033] In various embodiments, each sensor node 102 deployed in any given space may have a unique number (e.g., MAC address) assigned to it. The system uses the unique number to create a structured network with the numbered sensor nodes 102. As shown in FIG. 6, each sensor node may cover a different area and carry this unique number, which is accessed by users in both the digital and also physical environments. In particular, the unique number is clearly printed on the sensor node's enclosure 105 and is declared in the user's screen when the user is deploying the sensor node. In this way, users can identify whether the location of their physical sensor node matches the digital representation in the electronic space they created in the deployment application.

[0034] After a sensor node 102 is added and configured in the system as described in Figures 7 and 8B, the sensor node 102 creates a profile. The user is prompted to either scan a QR code located on the sensor, thus automatically registering a digital representation of the physical sensor, or manually type in the MAC address of the sensor. The sensor node 102 profile may include the sensor node 102 height, MAC address, relative location in space, surrounding objects, and / or context information. The system determines the extent of coverage for each sensor node based on the height of the sensor node 102, which the user enters as information into its profile. Some examples of context information may include the name of the room in which the sensor is located, a name for the sensor itself (if the user wishes to assign one), and a number assigned to the sensor. The user may upload a file about the surrounding environment, as shown in Figure 8A. Such a file may include, for example, PDF, JPG, PNG, 3DM, OBJ, FBX, STL, and / or SKP. The ambient information may include furniture in the sensor node 102's field, architectural layout around the sensor node 102's field, etc. The user may have the discretion to add ambient objects in the space. The sensor node may not register the ambient objects, however, the ambient objects may provide a richer visual context for the user's own personal use. The sensor node 102 profile may be stored in the user's profile in the database 360.

[0035] The thermopile sensor module 115 may project the temperature of the object onto a grid. The grid may be an 8 pixel by 8 pixel grid, a 16 pixel by 16 pixel grid, or a 32 pixel by 32 pixel grid (64 pixels, 256 pixels, or 1,024 pixels, respectively). The thermopile sensor module 115 may be tuned to detect a specific heat spectrum and allow detection of an object (e.g., a human body) with a standard temperature. The average normal body temperature is generally accepted as 98.6°F (37°C). However, normal body temperature may have a wide range from 97°F (36.1°C) to 99°F (37.2°C). Higher temperatures most often indicate an infection or illness. The system may detect such differences in temperature because the sensor module 115 may have an accuracy of 0.5°C. If multiple human bodies are present in the same area, the thermopile sensor module 115 captures and processes each body as a distinctly different heat source. In particular, the system includes a calibration process (an example user interface is shown in FIG. 10) built into the 3D front end to avoid overlapping temperature readings from different bodies.

[0036] As part of the calibration process (an exemplary user interface is shown in FIG. 10), the user is asked to step out of coverage of all sensor modules 115 through the application interface so that the system can automatically adjust the sensitivity of the sensor modules 115. Starting from maximum sensitivity, the system gradually decreases its sensitivity until there is no more high frequency detection of noise. Absolute rejection of noise allows detection of the body as a distinct heat source, and subsequently detection of two heat sources, such as a human body, as distinct and separate entities. For spatial overlap readings between the "field of view" of two sensor modules 115, during the calibration process the system recognizes the overlap area between the two sensor modules 115 and averages the common detections between the two sensor modules 115. If there is overlap detected between more than two sensor modules 115, the system averages the overlap pairwise, sequentially. For example, for overlap between sensor modules 115 A, B, and C, the system averages A and B, then proceeds to average the results of AB and C.

[0037] If the automatic calibration fails during the calibration process (an exemplary user interface is shown in FIG. 10), the system automatically generates digital paths between the deployed sensor nodes and prompts the user to physically stand under each sensor node. In doing so, the sensor nodes 102 detect the user's movement, and the detection becomes visible inside the application. If successful, the user is prompted to follow the path, thus completing the calibration for each sensor node 102 of the network. If any sensor does not respond as mentioned above, the user is prompted to digitally manipulate the sensitivity of the sensor module 115 by sliding a digital bar and adjusting the sensitivity level of that sensor node accordingly. More specifically, during troubleshooting of the calibration process, the user is prompted to stand under a physical sensor, one at a time, in each of the four corners of its field of view. At each corner, named by default in the app as A, B, C, and D, the user is asked to remain as long as the system detects their presence and successfully prints this on a digital replica of the sensor. Once a location is detected, the application asks the user which of the four possible corners they are attempting to mark.

[0038] In various embodiments, the thermopile sensor module 115 may detect an array of temperature readings. In addition to detecting the temperature of the living organism based on the pixel values, the thermopile sensor module 115 may also obtain the temperature of the environment in which the sensor module 115 is located. The system uses both local and global information (both each pixel individually and the network, i.e., all the sensor modules 115 forming the pixel as a whole) to determine the background temperature field. The thermopile sensor module 115 may obtain an independent temperature measurement of the sensor node 102 itself. The temperature of the sensor module 115 may be obtained using an on-board thermocouple. The system may use the temperature of the sensor node 102 and / or the temperature of the sensor module 115 to provide an assessment of the temperature profile of the space being monitored. The on-board thermocouple measurement itself measures the temperature of the space at the sensor location. The system uses bilinear interpolation to estimate the temperature between the sensor nodes 102 and / or the sensor modules 115 in the space and to approximate the temperature distribution. Also, in various embodiments, the system may measure and capture the temperature of the environment multiple times throughout the day to reduce the detrimental effects of maintaining a fixed background temperature field on threshold calculations, thus increasing overall detection accuracy in real-world scenarios where the environmental temperature is dynamic.

[0039] In various embodiments, multiple sensor nodes 102 can provide information in real-time and help produce real-time location, trajectory, and / or behavior analysis of human activities. By employing multiple sensor nodes 102, based on the density of the network, the system can infer the trajectory of any moving object detected by the sensor nodes 102. As mentioned above, the thermopile sensor module 115 inside the sensor node is designed to measure temperature from a distance by detecting the infrared (IR) energy of the object. The higher the temperature, the more IR energy is emitted. The thermopile sensor module 115, consisting of a small thermocouple on a silicon chip, absorbs the energy and produces an output signal. The output signal is a small voltage, which is proportional to the surface temperature of the IR emitting object in front of the sensor. Each thermopile sensor module 115 has 64 thermopiles, each of which is sensitive to the IR energy emitted by the object. To determine the trajectory, in various embodiments, each sensor module 115 divides the area captured by the sensor module 115 into a number of pixels organized in a rectangular grid in a direction aligned with 64 thermopiles, each of which is associated with an 8×8 portion of one of the aforementioned grids. The system monitors the successive changes in the temperature of sequential pixels. The system determines that such successive changes indicate the movement of an organism. The system logs such movements as the formation of a trajectory in space. The more nodes in the network, the more accurate the inference about the trajectory will be, since the detected trajectory will not be interrupted by "blind spots".

[0040] The calculation engine analyzes human behavior and trajectories. For example, for occupancy control, the system may calculate the total number of people in a space and compare it to established occupancy requirements. The system identifies all of the heat sources in the space monitored by the sensor module 115 and adds up the number of all heat sources generated by people.

[0041] With regard to occupant temperature screening, the system may detect the presence of a person by capturing the person's body heat. Temperature screening may include automatic adjustments to the sensitivity of the sensor module 115 once such detection is detected. Note that temperature screening may differ from body location detection. Body temperature screening refers to detecting an elevated body temperature of a detected person such that the sensitivity requirements are higher than body location detection alone.

[0042] With regard to monitoring the occupant's body temperature, the system 100 may be able to obtain the user's body temperature within a near field of one (1) meter from the sensor node 102 through reading the temperature of an area near the eye socket using a more detailed 32×32 grid in the sensor module 115. For the system to locate the eye socket, the user may be asked to gaze directly at the sensor node 102, allowing the sensor module 115 to detect the highest temperature pixel.

[0043] Regarding analyzing the occupant movement speed, the system may log the movement of a person under the network of sensor nodes 102. A series of "waypoints" are produced according to time. The system uses the distance traveled and the time taken to travel the distance based on the waypoint information to calculate the user's movement speed.

[0044] With respect to calculating the total calories burned based on the occupant's movements, the user inputs information such as the occupant's weight, gender, and age into the system through interface 370. The system may use the movement speed and captured distance (as mentioned above) to calculate the approximate calories burned during the time of the captured movement.

[0045] Behavioral analysis stems from the fact that by overlaying a structured network with real space, the captured data becomes contextualized. For example, the system can understand the shopping behavior of a mobile object by cross-referencing the actual trajectory and dwell time captured by the sensor node 102 with architectural plans that carry information about specific products and aisle locations, as described in Figures 5 and 6. In particular, in various embodiments, the user interface 370 allows the user to create a three-dimensional representation of the space, as shown in Figures 5, 6, and 9. For example, a grocery store owner may log information about products or ice cream freezer locations by naming each sensor node 102 or "tagging" it with the specific products located within its field. Thus, if sensor node #1 detects IR energy for 30 seconds, the system determines that a person has remained within sensor node #1's field for 30 seconds. If sensor node #1 is tagged as being in front of the ice cream freezer, the system will provide data that a person remained in front of the ice cream freezer for 30 seconds. Examples of system outputs are shown in Figures 4A and 4B.

[0046] In various embodiments, as shown in Figures 1A and 1B, each of the multiple sensor nodes 102 may interface with a module. The module may include, for example, an HTPA 32D module or an HTPA 16D module. The module may be a wireless module. The module may be wireless. The module may be a hardware module.

[0047] The sensor node 102 may include a switch 120 (e.g., ALPS) that controls power to the sensor node 102. The switch 120 may allow the manufacturer of the system to turn off the power to the sensor node 102 and conserve the module's battery 130 throughout its transportation or shipment from the manufacturer to the client. After the sensor node 102 is delivered to the client, the system may be deployed with the switch 120 to the sensor node 102 turned on and left on. If the client shuts down the store or the system for a period of time, the client may use the switch 120 to shut off the sensor node 102 and conserve battery life. An LED 125 on the sensor node 102 indicates system status, such as, for example, on and off mode.

[0048] A general data flow according to various embodiments is described in FIG. 2. The sensor node 102 may receive raw thermal data from the environment (step 205). The raw thermal data is compressed to create compressed thermal data (step 210). The gateway 135 receives the compressed thermal data from the sensor node 102. The gateway 135 decompresses the compressed thermal data to create decompressed thermal data (step 215). The cloud computing module 350 on the server receives the decompressed thermal data and creates detection data (step 220). A post-processing computing module on the server receives the detection data from the cloud computing module 350. The post-processing computing module processes the detection data and creates post-processed detection data (step 225). The post-processing computing module sends the post-processed detection data to a database 360. The database 360 ​​uses the post-processed detection data to create time-series detection result data (step 230). The system applies context analysis algorithms to the time-series detection result data to create analysis results (step 235). The system applies API services 380 and client apps to the analysis results to obtain 3D / 2D visualization of the data (step 240).

[0049] A general system architecture, including further details on data flow, according to various embodiments is described in FIG. 3. In hardware, the sensor node 102 obtains raw data and then performs edge compression (step 305) and / or edge calculation (step 310) to create Message Queuing Telemetry Transport (MQTT) raw data. A cloud computing module 350 on the server receives MQTT raw data topic 1 from the sensor module 115 via the gateway 135. The cloud computing module 350 applies data stitching and decompression (step 315) to MQTT raw data topic 1 to create MQTT raw data topic 2. The cloud computing module 350 applies core algorithms (step 320) to MQTT raw data topic 2 to create MQTT result topic 1. The cloud computing module 350 applies world coordinate remap (step 325) to MQTT result topic 1 to create MQTT result topic 2. Cloud computing module 350 sends MQTT raw data topic 1, MQTT raw data topic 2, MQTT result topic 1, and MQTT result topic 2 to database 360. Influx DB receives the data. Database 360 ​​applies context analysis (step 330) to the data via context analysis algorithms to create context results (analysis results). The context results are stored in Dynamo DB. Dynamo DB also stores sensor node 102 profiles. Database 360 ​​may apply additional context analysis in response to updates or additional settings to the sensor module 115 profile. API 380 retrieves the context results from database 360. The API applies real-time raw data, real-time detection, historical raw data, historical detection, historical occupancy, historical traffic, and / or historical duration to the data (step 335). The API sends the results to a user interface 370 (e.g., on a client device).The user interface 370 provides visualization (step 340) (e.g., FIGS. 4A and 4B). The user interface 370 also provides a configuration interface (step 345). The configuration interface may provide updates to the sensor module 115 profile. The user interface 370 also provides login functionality (step 350). The login functionality may include AuthO / Firebase.

[0050] More specifically, the sensor module 115 may collect sensor module 115 data, pre-process the data, and / or transmit the collected sensor module 115 data to the gateway 135. The module may include an on-board microprocessor 140. Raw data from the sensor module 115 may be stored in RAM of the microprocessor 140. The RAM serves as temporary memory for the system. The microprocessor 140 is configured to pre-process the raw data by eliminating outliers in the raw data.

[0051] In particular, the microprocessor 140 applies a defined statistical procedure to the raw data to obtain processed data. In various embodiments, the module uses firmware software to perform pre-processing. The firmware software statistically determines outliers of the temperature readings. Outliers may be defined by normalizing the data, for example, by subtracting each pixel value from the average value of the frame. The result is divided by the standard deviation of the frame. Pixel values ​​that are above or below three times the standard deviation are removed and replaced using a bilinear interpolation technique, i.e., with an interpolated product of adjacent values. Pixel values ​​are replaced instead of removed so that the input detection is similar before and after the procedure. This technique helps repair even minor data issues that may be caused due to potential shortcomings in the sensor module 115 data quality. The combination of firmware software, circuit design, and drivers allows the system to launch algorithms and determine a "region of interest" on each of the data frames to represent human activity under the sensor module 115 view. The region of interest is not centered around a pixel with a certain temperature, but rather a pixel with a different temperature (higher or sometimes lower) relative to its surrounding pixels. The region of interest is then used to compress the processed data and prepare the compressed data for wireless transmission.

[0052] The system may include a rolling cache of 10 data frames to perform pre-processing. More specifically, firmware in microprocessor 140 may use the last 10 data frames of captured data to perform pre-processing and post-processing procedures. The system may only process a subset of the data due to the limited amount of RAM memory on-board (e.g., 8kb for applications).

[0053] Data passing through the gateway 135 may be uploaded to the cloud computing module 350 on the server. The gateway 135 may be powered by any power source. In various embodiments, the gateway 135 is powered by a 110V outlet. The gateway 135 includes a module for connecting to a network (e.g., the Internet) via Ethernet, wifi, and / or cellular connection. Thus, the gateway 135 may upload data to any database 360, server, and / or cloud. The gateway 135 sends pre-processed and compressed data to a computation engine in the cloud computing module 350, which in turn outputs the results to the database 360s. The gateway 135 performs management functions such as software updates and pulls operational commands from the server to command the modules to turn the sensor module 115 on and off, change sampling frequency, etc.

[0054] In various embodiments, the gateway 135 will capture the compressed raw data upon transmission and send it to an algorithm running on a processor (e.g., Raspberry Pi 4, model BCM2711) which will then forward the information to a server (e.g., cloud computing) for further processing. Processing of the data on the server includes decoding the compressed raw data, normalizing the sensor module 115 temperature data per each sensor module 115 firmware and environment settings, detecting objects, classifying objects, spatially transforming for world coordinate system positioning and fusion, multi-sensor module 115 data fusion, object tracking and trajectory generation, cleaning outlier pixel-level readings, and other post-processing.

[0055] In various embodiments, the processing steps operate with decompressed raw data. Decoding the compressed raw data optimizes data transmission and battery 130 consumption levels. Normalization of the sensor node 102 temperatures to appropriate temperature ranges also allows the processing steps to adapt to various qualitative and environmental differences (expected from sensor nodes 102 located at different spots in space).

[0056] One of the core processing steps of the computational engine of the cloud computing module 350 is object detection and classification. This processing step detects the location of objects of interest in the frame and classifies the objects into different categories of people or objects (e.g., laptops, coffee mugs, etc.). A spatial transformation from local to world coordinate system makes the analysis "context-aware". Using spatial transformation, the system may compare and cross-reference the spatial sensor module 115 coverage with the actual floor plan and 3D model of the space. Multi-sensor module 115 data fusion integrates data in case of missing information or overlapping coverage between multiple sensor modules 115. As mentioned above, using various algorithms, object tracking and trajectory generation differentiates multiple people from each other throughout time. Object tracking and trajectory generation provides a set of trajectories resulting from detected objects and people. The system uses such trajectories to determine behavior analysis (e.g., dwell location and duration), movement speed, and direction. A post-processing step resolves any minor discrepancies in the detection and tracking algorithms. For example, when a gap or detection exists in the trajectory, a post-processing step helps stitch the information together and correct any broken trajectories.

[0057] The system may use the heatic Application Protocol Interface (API), which may be located in the API layer in the system architecture, as described in FIG. 3. The API hosts real-time and historical people count data for the space that is enhanced with the sensor module 115 solution. Built on REST, the API returns JSON responses and supports cross-origin resource sharing. The solution employs standard HTTP verbs to perform CRUD operations, while for error indication purposes the API returns standard HTTP response codes. In addition, namespaces are used to implement API versioning, while all API requests are authenticated using token authentication. An API token, found on the dashboard, is used to authenticate all API endpoints.

[0058] This token may be included in the Authorization HTTP header, prepended with the string "Token" with a single space separating the two strings. A 403 error message will be generated if the proper Authorization header is not included in an API call. The Authorization HTTP header will contain the Token YOUR_API_TOKEN. The endpoint uses standard HTTP error codes. The response includes any additional information about the error.

[0059] The API lists low level "sensor module 115 events" for a sensor module 115 and a period of time. A timestamp and trajectory for that sensor module 115 are included in each sensor module 115 event. It is not necessary that the trajectory be equal in any direction in space (e.g., entrance or exit). This call should only be used to test sensor module 115 performance.

[0060] The API provides information about the total number of entries into a specific space on a per day basis over a one week duration. An analysis object with the data and total number of entries over that interval is nested within the interval object of each result. This call may be used to find the number of people who visited a space on different days of a week.

[0061] The API documents, counts, and lists all individual exits from a space of interest over the course of an entire day (or any 24 hour period). Each result carries a timestamp and a direction (e.g., -1). This call is used to find when people leave a space.

[0062] The API provides information about the current and historical wait times at the entrance of a specific space at any given time during the day. An analysis object with the data over that interval and the total estimated wait duration is nested within the interval object of each result. This call is used to find the number of people who were in line to get into a space at different times of day.

[0063] A webhook subscription allows for the receipt of callbacks to a specified endpoint on the server. A webhook may be triggered after every event received from one of the sensor modules 115 for each space in which the event occurred. The system may create a webhook, get a webhook, update a webhook, or delete a webhook. When a webhook is received, the JSON data will be similar to the space and sensor module 115 event in the previous section. It will have additional information, namely the current count for the associated space and the ID of the space itself. The direction field will be 1 for an entry and -1 for an exit. If applicable, additional headers will be configured for the webhook and will be included with the POST request. An example of received webhook data may be a single event occurring on a path connecting two spaces.

[0064] In various embodiments, the system may include one or more tools to help facilitate deployment, configure software and hardware, provide more accurate detection, create virtual representations, visualize human movement, test devices, and / or troubleshoot devices. The system may include any type of software and / or hardware, such as, for example, one or more apps, GUIs, dashboards, APIs, platforms, tools, web-based tools, and / or algorithms. The system may be in the form of downloadable software. For example, the software may be in the form of an app downloaded from a website that may be used on a desktop or laptop. The software may also include a web application that may be accessed via a browser. Such a web application may be device independent and adaptive, such that the web application may be accessible on a desktop, laptop, or mobile device. The system may be obtained by license or subscription. One or more login credentials may be used to partially or fully access the system.

[0065] As used herein, a space may include the overall layout of an area, which may consist of one or more rooms. System functionality may affect different spaces separately. The system may associate multiple rooms within a space. A room may include any walled portion of a space (e.g., a conference room) or an open area within a space (e.g., a hot desk area or a hallway). A headcount may include the number of people moving in and out of a space or room within a given time range. An occupancy rate may include the number of people inside a room or space at a given time. A fixed object may include furniture (e.g., chairs, desks, etc.) or equipment (e.g., washing machine, stove, etc.).

[0066] In general, in various embodiments, the system may plan the deployment, for example, by visualizing sensor placement, visualizing coverage within a space using a 3D drag-and-drop interface, and / or understanding the number of sensors and / or hives that may be optimal for a room or space. The system may enable more accurate detection, for example, by analyzing the spatial context to distinguish between humans and inanimate objects (and other confounding factors). The system may obtain the spatial context, for example, by receiving inputs about the layout of the space, 3D furniture, rooms, and signs / tags. The analysis of the spatial context may involve artificial intelligence and / or machine learning. Using the spatial context, the AI ​​is used to learn about the space so that it can accurately identify human presence, behavior, posture, and other specific activities. The system may create a virtual representation of the real place (e.g., across any of the dashboards, settings apps, and any other applications that use algorithms and APIs to visualize spatial data). The virtual representation may be based on a tool that receives signs, tags, and / or names for sensors, rooms, and spaces. The virtual representation may be represented as a unique identifier in the dashboard and / or API.

[0067] In various embodiments, the system may visualize human movement, for example, by showing current and previous frames (e.g., in the form of figures such as dots), showing and listing location coordinates in the context of sensors, virtual space layouts, and virtual fixtures, showing the person's trajectory, and / or showing the person's posture (e.g., standing, sitting, lying down). The system may test and troubleshoot the device, for example, by showing the user what the sensors are detecting. The user can then verify that the actual object types and locations correlate to the visual representation. The system may show what the sensors are detecting in a real-time and / or frame-by-frame representation of the human's presence and movement in the space. The system may also indicate when the sensors are online, offline, connected, and / or disconnected.

[0068] In various embodiments, the system may include functionality for creating a space layout. Creating a space layout may involve adding a space and adding a space name. The system may then store the space with its name. The system provides functionality for a user to create and manage multiple spaces. Having several separate spaces may be useful when monitoring multiple floors in one building (e.g., 1st floor, 2nd floor), a facility with multiple individual rooms (e.g., senior living apartments), or multiple facilities in separate physical locations (e.g., a lab in Boston, a lab in San Francisco). As part of the setup, in various embodiments, the system may provide the ability to one or more of: rename the space, add auto-match, add visualization smoothing, add display local detection, add toolbars (e.g., main, side, etc.), add fixtures from a library (e.g., a piece of furniture from a furniture library), or return to a project library. The main toolbar may include functionality related to rooms, sensors, hives, languages, and / or save. Exemplary side toolbar functions may include 2D or 3D, show or hide sensors, show or hide rooms, show or hide fixtures, etc. Although the toolbars and functions may be described as being located on the main toolbar or the side toolbar, any of the functionality may be associated with any toolbar.

[0069] In various embodiments, the system may include functionality for adding rooms to match or resemble a floor layout. The system may present a control panel (e.g., in response to selecting a room). The control panel may allow changing dimensions, tagging certain locations or features, and / or selecting a border color for each room. In response to selecting a room icon, the system may add one or more rooms to the space. In response to automatic matching being activated, a moved object (e.g., a fixture, sensor, or room) may be automatically aligned to the edge of a nearby object of the same type. A chair may be automatically moved next to another object, for example, making it easier and quicker for a user to arrange objects in an orderly and tidy manner to match a floor plan. The system may determine that the moved object is the same type of object as an existing object based on similar identifiers or labels associated with each of the objects. In response to automatic matching being deactivated, the user is free to move objects that cannot be matched with similar objects in increments.

[0070] In various embodiments, the system may include functionality for adding fixtures to the virtual space in the GUI (which may correlate to fixtures present in the physical space). Fixtures may include, for example, furniture or equipment that the user may add to the space. Fixtures may allow the user to distinguish between rooms and allow the user to contextualize the movements seen on the screen. The system may allow the user to add fixtures by selecting one of the furniture or equipment icons and then dragging and dropping the fixture to a specific location in a different room. The system also provides functionality for the user to virtually select a fixture and then delete or rotate the fixture using a panel control. The system also provides functionality for the user to virtually adjust the size or location of the fixture. In various embodiments, the user may input the specific coordinates of the furniture, so the system knows the dimensions and location of the furniture. Also, as the user moves a fixture, the system may show the distance between the center point of the fixture and each of the four walls of the room in which it is installed. The coordinates of the furniture relative to the entire space may be stored in the system through an API. In this way, the user can pull this information from the backend as needed. A user can add as many fixtures as needed in a space or room. A user may layer fixtures or furniture over other fixtures or furniture. Also, if a physical table is particularly large, a user may use multiple virtual tables to match its size. The system encodes, recognizes, stores, and takes into account the presence and coordinates of fixtures. The system uses such data when determining whether a detection is human and whether the detection should be counted with respect to occupancy or number of people in a room or space. The system includes functionality (e.g., using APIs and algorithms) to encode each fixture with a fixture type (e.g., table, door, etc.). A user may select the fixture and fixture type from an icon.The API may store the fixture and its coordinates on the system so that algorithms may use this information to identify detection and behavior. In response to a user placement, the system records the fixture's center point xy coordinates, fixture type, and rotation from the center point in degrees (e.g., 0, 90, 180, 270). Rotation may include an action to rotate an object (room, sensor, fixture) 0, 90, 180, 270 degrees from its center point. Rotation may be implemented by selecting a circular arrow to rotate the object. Fixtures in the system may be set to a default pointing orientation that may differ from the real furniture in the room. The user can rotate the virtual furniture model in four different directions (i.e., 0, 90, 180, 270 degrees from its default orientation) around the center point of the model.

[0071] The system may display when people are located on, around, or passing by a piece of furniture. The system stores names or icons associated with different fixtures. Such names or icons include factors or rules for the system to consider when analyzing the fixture. For example, a bed icon may include a rule that a person may be located on a bed, while a table icon may include a rule that a person may not be located on a table. Other examples of fixtures with rules that a person may not be located on include a table, a counter, a stove, a refrigerator, a dishwasher, a sink, a radiator, a counter, and / or a washing machine. Other examples of fixtures with rules that a person may be located on or in include a bed, a couch, a chair, a toilet, and / or a shower. The system may infer an activity based on a person being located by a piece of furniture for a certain amount of time. For example, if the system detects a person in the vicinity of a TV for an hour, the system may infer that the person liked the program playing on the TV. This location and inference information also provides useful contextual information to the algorithm to enable the system to infer daily activities and enable more accurate human detection. For example, with respect to daily activities, if a detection of an object (e.g., represented by a purple sphere) is on or within the outline of a bed fixture, the system may infer that a human is sleeping. The system may then infer the person's sleeping time by determining the length the person is located on the bed fixture. As another example, for more accurate human detection, the system may detect both humans and non-human heat sources such as stove tops and laptops. If a detection of a heat source is found to be located in the center of a fixture such as a table, the system may recognize that this is not a human. Thus, the system may not count this detection as part of occupancy data (e.g., that a user may receive via an API or dashboard).The system may include functionality to activate or deactivate a "Show local detections" option to view or hide detections (e.g., purple spheres).

[0072] In various embodiments, the system may include functionality for "visualization smoothing." Deactivating "visualization smoothing" causes the detection sphere to be shown exactly the way it is detected, frame by frame, on specific coordinates. Activating "visualization smoothing" causes the frame by frame movement of the purple sphere to be shown with a smooth continuous animation. More specifically, when a sensor detects an object in the physical space, the system may create an "agent" (e.g., a purple sphere) that appears on the corresponding location in the virtual space. The system may also assign a "lifespan" to that agent. A "lifespan" may be the length of time that the purple sphere appears in the virtual space to indicate a detection. In various embodiments, such a lifespan may be set to last for 300 milliseconds, matching (or similar to) the interval at which the sensor transmits a new detection. The system uses the new detection at the new location in the physical world to update the placement of the purple sphere in the virtual space. As mentioned, the system may display the purple sphere on a frame by frame or with visualization smoothing. Although the purple sphere may appear to be blinking, the sphere may actually represent a real-time detection captured by the sensor every 300 milliseconds. The blinking effect is due to the fact that the purple sphere goes from translucent to transparent in its 300 millisecond lifespan. If the sphere is more opaque, the detection may be more recent. If a person is standing stationary under the actual sensor, the purple sphere will appear to be blinking in place. If a person is moving under the actual sensor and visualization smoothing is not activated, the system may show the trajectory of the purple sphere. For example, each purple sphere may go from opaque to transparent in succession at each detection coordinate (or pixel) every 300 milliseconds. When visualization smoothing is activated, only one purple sphere appears on the screen and appears to move linearly. During the 300 millisecond lifespan, the system searches for the next detection within a one-foot radius and "recharges" its lifespan for another 300 milliseconds. This means that the spheres constantly appear and move from one spot to another according to the detection coordinates (pixels).

[0073] In various embodiments, the system may include functionality to add one or more hive and / or heatic sensors. Data from outside the room may still be stored via the API, but the outside room data may not be shown on the dashboard. The dashboard shows the room-specific activity and occupancy in the space. Thus, the system instructs the user to place sensors and objects in the room. The hive may provide gateway functionality to connect and transmit data from the heatic sensor to a storage location (e.g., the cloud). For example, the hive may connect to a certain number of sensors (e.g., up to 12 sensors), and then additional hives may be needed for the additional sensors. For easy deployment, each hive may be pre-configured with a set of sensors (e.g., sensor IDs are loaded into the database for the hive as sensor data). The pre-configuration process may include two parts. The sensors may be set to the same NetID as the hive. This is what allows the sensors to connect to each other. The hive may get the sensor MAC address and sensor mode programmed into the configuration file. This allows the hive to properly manage the sensor frame rate. The system may allow a user to add one or more hives to a space. The system may include functionality to add a hive to a space by scanning a code (e.g., a QR code) or by typing in a hive ID (e.g., found on a sticker under the hive). During the process in which hive data is added to the system, the system also receives preconfigured sensor data (because the hive data was preconfigured to include sensor data). The system uses the sensor data to display a "sensor" icon for each hive. The system may indicate sensors from different hives, as sensors may be color-coded to indicate that the sensor belongs to a particular group and hive. Specific implementations may vary, but generally may be visually identifiable throughout the system.In response to receiving a selection of a sensor icon for a hive, all of the sensors associated with that hive are displayed so that each sensor can be added to the space. The user may drag and drop each virtual sensor to a location in a room to record room occupancy, or onto a doorway to record the number of people moving in and out of the doorway. Each sensor may be unique and may be identified with a unique address (e.g., MAC address). Thus, the user should place the virtual sensor in the same location, orientation, and room as the corresponding physical sensor.

[0074] In various embodiments, the system may include functionality for setting up and / or calibrating virtual heatic sensors. Each virtual heatic sensor may appear on the display (e.g., as a square). In response to a virtual sensor being turned off, the virtual sensor may appear in some manner (e.g., a black square) to indicate that the real sensor is not detecting anything. In response to being turned on, the virtual sensor appears as a grid (e.g., an 8×8 grid of 64 squares). Each of the 64 squares may represent a pixel, and the color of each pixel may represent the temperature that the real sensor detects at that point in the grid. The color of each pixel may include a range of shades to indicate temperature levels. For example, the range of shades may be from yellow (lower temperature) to red (higher temperature).

[0075] In response to receiving a selection of a virtual sensor, the system displays a control panel (e.g., on the left side of the display). Using the control panel, the system may allow the user to set the virtual sensor height. For example, an aisle in a grocery store may be important to be within the field of view, but a guard room may be outside the field of view. The optimal (and preferred maximum) height of the actual sensor is about 3.2 meters. Such a height may provide the maximum coverage and optimal resolution required for human detection. The optimal resolution may be based on the ability of the algorithm to detect human presence from the heat map image resulting from the sensor. The optimal resolution refers to the resolution at which the system may accurately and reliably detect human presence. The virtual sensor height corresponds to the ceiling height or height on the wall where the actual sensor would be mounted. The higher the sensor above the floor, the greater the floor coverage of the sensor. The lower the sensor relative to the floor, the lesser the floor coverage of the sensor. Based on the height, the system determines the amount of floor space the sensor is monitoring (covering).

[0076] This system uses the formula (90% x 2 x tan(30°) x height) 2 A sensor height of 110 inches (2.8 m) may provide an effective coverage width of 106 inches (2.7 m), a sensor height of 102 inches (2.6 m) may provide an effective coverage width of 78 inches (2.0 m), a sensor height of 95 inches (2.4 m) may provide an effective coverage width of 63 inches (1.6 m), and a sensor height of 87 inches (2.2 m) may provide an effective coverage width of 56 inches (1.4 m).

[0077] In various embodiments, the system may also allow users to set virtual sensor orientations to match the way real sensors are physically arranged to ensure accurate representation between the virtual world (e.g., in the Settings app) and the physical world. Sensor orientations in both the real and virtual worlds should be similar so that visual detection also matches. For example, a person standing in the northeast corner of a room should appear on the northeast corner of the corresponding virtual sensor pixel in the Settings app. To help match sensor orientations, a physical sensor may include an arrow (e.g., on its mounting plate). When a user adds a physical sensor to the Settings app as a virtual sensor, the user matches the orientation of the virtual sensor arrow to that of the physical sensor arrow by rotating the virtual sensor.

[0078] In various embodiments, the system may include functionality for viewing detections. "Detection" refers primarily to the detection of a person's presence. The actual sensor may capture a heat map of an area at, for example, 3-5 frames per second. The system may detect a human presence by identifying areas of the heat map with approximately the body temperature (or "thermal signature") of a human. The system may represent the detection of a person on the display (e.g., as a purple sphere). The average normal body temperature for a person is generally accepted as 98.6°F (37°C). However, normal body temperature may have a wide range from 97°F (36.1°C) to 99°F (37.2°C). A temperature above 100.4°F (38°C) may imply that a person may have a fever caused by an infection or illness.

[0079] In various embodiments, the detection process may include sensitivity adjustment. The system may receive data about the temperature within a space or room based on a thermometer located within the physical space or room. Sensitivity adjustment may involve improving the system's ability to detect the presence of a human in an environment with a temperature closer to that of a human body. Detection is improved by changing a parameter involving the temperature difference between the detection (human temperature) and the surrounding environment. Increasing sensitivity may involve minimizing this temperature difference so that the system may more easily detect the presence of a human in, for example, a very warm climate. In other words, in a colder climate of 65 degrees, the system may more easily determine that any object that is 30 degrees or more warmer than a typical room temperature may be a human. However, in a warmer climate of 96 degrees, the system may need to detect an object with a temperature difference of 2-3 degrees warmer than room temperature to consider the object a human.

[0080] This detection information may be further processed by system algorithms to double-check and ensure that the final data sent to the API and dashboard is accurate. Such further processing may include additional criteria to filter out detections that do not behave like humans. The system may filter out any objects with temperatures lower than the range of human body temperatures (e.g., 97°F (36.1°C) to 99°F (37.2°C)). Filtering may include detections that do not move at all (e.g., appliances such as stoves) or stationary detections that appear to be on fixed objects where a human is not expected to be on (e.g., in the center of a table). For example, the system may determine that a heat map shown on a table is more likely to indicate a laptop. The system may also store the coordinates around each fixed object so that detections of objects with coordinates that overlap with the coordinates of any fixed object are not counted. In other words, the coordinates around the fixed object are blacklisted to not count any detections within those coordinates.

[0081] In various embodiments, the detection may include a number of people. A sensor (a people sensor) that determines the number of people from people entering and leaving the room may be disposed above the access doorway of the room (e.g., on a wall facing the inside of the room). The people sensor may use data associated with a virtual threshold (or door line), which may be a certain distance from the door. The people sensor may only count detections of people crossing the door line. For example, "inside" is when a person crosses the door line from left to right, while "outside" is when a person crosses the door line from right to left. The door line, for example, reduces false readings from people who may put their head inside the door to see what is inside the room, but never fully enter the room.

[0082] In various embodiments, the system may include functionality for 2D or 3D view configuration. In response to selection of the 2D / 3D button, the system may display all (or any portion) of the entire space in 2D or 3D. A 3D view may provide a more "realistic" spatial context to users who are not familiar with floor plans. Such a view may also provide a more intuitive way to understand the space, detections, and data. In various embodiments, the system may include functionality for showing or hiding various features, such as sensors, rooms, or fixtures. The system may include functionality for editing spaces, managing spaces, different viewing modes for visually conveying data, and the like.

[0083] In various embodiments, the system may visually convey data about foot traffic and / or dwell time over a period of time. Foot traffic may be conveyed based on the number of people proceeding "in" and "out" of a given doorway (or across a door line) and into a room or space. Foot traffic may also be conveyed by a walking path or trajectory line of people's movement within a space. The system may include a people count view, where the number of in and out may be shown on a virtual layout. The system may include a trajectory view, which shows multiple detections over a period of time, thereby forming a stream of detections in a display. The stream of detections may be used to determine a path of foot traffic. The system may also show a linear trace of movement, where the system creates a line through the multiple detections, thereby creating a line that represents the foot traffic path. Dwell time may be the amount of time a person spends in a room or at a set of coordinates in a space. Dwell time may be determined by measuring the amount of time a person is detected in an area. The system cannot determine if this is the "same" person previously detected, but the system may infer or identify a unique detection (same person) by location and trajectory. Dwell time may be shown as a heat map, with darker colors indicating more time spent in a specific area and lighter colors indicating less time spent in an area.

[0084] As discussed above, traditional computer vision uses high-resolution images to develop human pose detection algorithms. High-resolution images may include, for example, recorded high-resolution video footage. Many of the existing algorithms detect key points (key points may be points on the human skeleton) to detect human pose from images, and key point detection requires a high-resolution camera system. Therefore, key point detection may not be feasible using reduced resolution systems. However, systems that can use reduced resolution images may be important to promote additional privacy.

[0085] In various embodiments, the system may provide functionality for posture detection by using a posture detection algorithm on reduced resolution images. For example, the system may indicate when a person is standing, sitting, lying, or falling. In general, the algorithm may include a sub-module for extracting a bounding box containing human pixels. The algorithm may also include a sub-module for detecting the posture of the person inside the bounding box. The algorithm may be a deep neural network learning-based algorithm that may be data-driven. The neural network may be a CNN (Convolutional Neural Network), which is a type of artificial neural network used in image recognition and processing that is specifically designed to process pixel data.

[0086] In general, demarcating the bounding box is one of the behaviors learned by the neural network. The neural network may be trained on images with bounding boxes, which may contain all pixels (or a subset thereof) that correspond to humans in the image. The neural network may use this information to predict where the bounding box should be on the image. More specifically, in various embodiments, the system may receive images of humans. The input images / frames may be low resolution, such as, for example, 8×8, 32×32, or 64×64 pixel images. For each frame, a human annotator may first observe the pose of the human (e.g., test subject) in the image, label the pose, and draw a bounding box around the people in the image. The human annotator may verify that humans are in the image by reviewing videos / images recorded at high resolution. However, such high resolution images may not be used to train the algorithm.

[0087] In various embodiments, the user may label the poses with an integer. The user may input the integer for a particular pose on the screen. The GUI may include a text box with a field that receives an integer for one or more poses. The system may associate the integer with a pose in a database. For example, label 0 may represent a sitting pose, label 1 may represent a standing pose, label 2 may represent a lying pose, etc. In various embodiments, the user may draw a bounding box on the screen (e.g., using any type of device that receives input on a GUI) around the human in the image such that the bounding box is stored as an (x,y) coordinate that the system can recognize. The system may use such labels and hand-drawn annotated data (e.g., bounding box) to train a learning algorithm. The learning algorithm may be trained, for example, using gradient descent-based training. Based on the training, the learning algorithm may learn how to automatically create bounding boxes, collect data from within the bounding boxes, and determine human poses from the collected data. The collected data used by the algorithm may be collected as the system boots up over time or during an initial calibration session. The system may obtain environmental data from the particular environment in which the sensors are deployed, and therefore the system may use the environmental data to adjust its algorithms based on the particular environment. Data from the environment may include parameters and / or variables from the environment, such as, for example, environmental temperature, indoor temperature, floor plan, non-human thermal objects, human gender, human age, height of the deployed sensor, human clothing, human weight, etc. Pixel data, thermal data, and / or environmental data collected from the environment may be used to train algorithms through machine learning or artificial intelligence.

[0088] Due to the low resolution, it may be difficult to determine the difference between image intensities between different pixels, and therefore the algorithm may not attempt to determine features based on image intensity. Rather, the algorithm may focus on the differentiating features of patterns from overhead heat signatures of human presence. CNNs may implicitly define what constitutes a "differentiating feature." Each layer of the network defines a "feature map" that is learned through stochastic gradient descent. The user may not be aware of the features that the network considers important. The system may find edges, curves, significant contrasts in adjacent pixel values, etc. However, as a nature of neural networks, the system may not have any (or very little) information about which features are important in differentiating poses.

[0089] As mentioned above, in various embodiments, the system may extract regions (e.g., represented as rectangles representing bounding boxes) that contain differentiating features of a human. The bounding boxes may limit the amount of data analyzed by the system, and the boxes force the algorithm to focus only on the outline of the human. The system may focus on the extracted bounding boxes and attempt to classify the human pose in this step of the algorithm. The system may extract human pose information on a frame-by-frame basis. A frame may be a single image captured by one of the thermal cameras.

[0090] In various embodiments, the system may calculate an "aggregated posture" to help smooth posture variations over a period of time across multiple frames. For example, the aggregate posture may be determined based on the mode of the set of postures collected over a given period of time (e.g., the postures that occur most frequently in the set). Although it is possible that not all postures in the set may be identical, the posture aggregation method creates a shared term. The shared term may be referred to as an "aggregated posture." For example, the system may obtain a posture shared term every 5 seconds. The aggregate posture may include, for example, sitting, standing, or lying. The system may determine an event to be a fall based on a change in posture or posture that lasts for a certain amount of time. For example, if an event consists of (i) an aggregate posture change from standing / sitting to lying and (ii) a lying posture that lasts for a certain amount of time (e.g., at least 30 seconds).

[0091] Using pattern recognition, fuzzy logic, artificial intelligence, and / or machine learning, the system may determine human postures based on similar thermal signatures. With regard to pattern recognition, in various embodiments, the system may extract differentiating features and / or patterns from frames to help identify postures that the frames contain. With regard to artificial intelligence, in various embodiments, based on collected human annotated data and / or frames, the system may automatically learn those patterns by using CNN. As described in FIG. 11, patterns from human thermal signatures may be different, and different patterns may cause the system to classify human postures differently. For example, FIG. 11A shows a thermal signature indicative of a standing position, FIG. 11B shows a thermal signature indicative of a sitting position, and FIG. 11C shows a thermal signature indicative of a lying position.

[0092] As described in FIG. 12, in various embodiments, the system may perform a pose inference process. The system may obtain an input image (e.g., an 8×8, 32×32, or 64×64 pixel image) from an infrared sensor (step 1205). Upper and / or lower resolution limits may be based on privacy concerns, sensor power consumption, data cost, data bandwidth, computation cost, and / or computation bandwidth. The system may apply a CNN to the input image (step 1210). The CNN may implicitly extract important information from the image by learning different filters at each subsequent layer of the network through stochastic gradient descent.

[0093] The system may create a vector of image features (step 1215). The vector may be an intermediate result output by the CNN consisting of learned image features. A transformer encoder / decoder may be used (step 1220) to convert the vector of image features into a box prediction (step 1225). A DEtection TRansformer (DETR) may use a traditional CNN backbone to learn a 2D representation of the input image. The model may flatten the image and complement the image with a positional encoding before passing it into the transformer encoder. The transformer decoder may then take as input a small fixed number of learned positional embeddings (e.g., object queries) and additionally contribute to the encoder output. The system may pass each decoder output embedding through a shared feedforward network (FFN) that predicts either a detection (e.g., class and bounding box) or a "no object" class.

[0094] In various embodiments, the system may include a box prediction, which may predict the size and location of a bounding box that contains the differentiating features of a human. The system may obtain the interior of the bounding box (step 1230). The system obtains the interior of the bounding box because anything that falls outside the bounding box may not be of interest to the system since the system is interested in identifying the human pose of the human. In various embodiments, the system applies a CNN to the interior of the bounding box (step 1235). More specifically, the CNN applied to the interior of the bounding box may implicitly learn features of the images it is trained on. These features may be invisible to a human user and may never be made explicit by the neural network. Based on the CNN, the system creates a pose prediction for the image (step 1240). More specifically, during the training phase of the model, the system may extract features for differentiating human poses. This is prior knowledge used during the inference / testing phase when the trained model is used to differentiate between various poses. The pose prediction may include, for example, sitting, standing, lying down, or any other pose or configuration of a human. The system may also detect other postures using a higher resolution (32x32 pixels or higher). For example, the system may perform better using a higher resolution so that the system may be able to differentiate between standing and sitting postures. The system may also differentiate activities such as exercising, dancing, running, eating, etc.

[0095] As described in FIG. 13, in various embodiments, the system may perform a fall detection process. The system may obtain images (step 1305). All of the images may be aggregated, for example, over 5 seconds. The system may benchmark the amount of time to aggregate the images. If the amount of time to aggregate the images is too short, the system may lose image details. If the amount of time to aggregate the images is too long, the system may obtain noisier data in the images. The system may not make any assumptions about the position of the person in the frame. The person may be stationary or moving. The system obtains data about the person's pose in each frame. The pose inference process described above is performed on the images (step 1310), as described in FIG. 12. The result of performing the pose inference algorithm on each frame is a pose prediction for that particular frame. After pose inference, the system may obtain a number of predicted "poses." FIG. 12 shows this as "poses collected over 5 seconds." After the "Posture Aggregation" module, the system may determine one "aggregated posture": either standing / sitting or lying down. The system may then use this information to further determine whether a fall has occurred.

[0096] In various embodiments, the system may perform posture aggregation (step 1315). More specifically, the system may determine an aggregated posture based on the mode of a set of postures collected over a given period of time. For example, the posture that occurs most frequently in the set. It is possible that not all postures in the set may be identical, but the posture aggregation method creates a shared term that is the aggregated posture. The system may determine the aggregated posture (step 1320). The aggregated posture may be determined to be standing or sitting (step 1325). The system may determine that the aggregated posture is lying (step 1330). If the collected posture is lying, the system may review a database of previous aggregated posture data to determine whether the aggregated posture at the previous timestamp was standing or sitting. The system may compare the posture to postures identified in previous frames. The frames may be ordered in time. The system may determine that the aggregated posture has changed from a standing or sitting position to a lying position and then the posture has remained in the lying position for at least 30 seconds (step 1335).

[0097] The system may store a timestamp associated with each of the various actions. The system may check the difference between the start timestamp and the current timestamp to determine if the time difference exceeds a threshold. The threshold may be predefined, pre-determined, dynamically adjusted, based on an algorithm, etc. In various embodiments, if the fall exceeds the threshold amount of time, the system issues a fall alert. The fall alert may be transmitted to any other device via any communication means. For example, the system may send a signal via the internet to an app on the relative's smartphone to notify the relative that a fall may have occurred. Along with the alert, the system may also provide data about the person, location, facility, health information, demographic profile, history of falls, etc.

[0098] Due to the nature of CNN, it may assign a certain type of pose. However, if the system is unsure of a pose or frame, the system may ignore some frames. For example, the system may use a 0.7 confidence score. A 0.7 confidence score may indicate that if the system predicts with 70% confidence, this is a definite fall. The system may also predict falls with more or less than 0.7 confidence.

[0099] The system may determine a potential fall (step 1340). For example, if a person is lying in bed or on a couch, the system may determine such an action as a "potential fall". However, lying on a bed or couch may be a normal action, and such an action may not be a real fall. Therefore, the system may perform post-processing on the potential fall (step 1345). In various embodiments, the post-processing may include filters such as, for example, occlusion of certain spatial locations (e.g., bed) where a fall cannot reasonably occur. Another filter may include a threshold for the confidence of the detection of a potential fall using the algorithm.

[0100] Based on the post-processing, the system may confirm the potential fall as a confirmed fall (step 1350). In particular, after using one or more filters, the system may confirm the fall and provide a notification of the confirmed fall. For example, in a user interface / software product, a user may create and place virtual furniture (e.g., bed, chair, closet, etc.) or occlusion areas in a settings app. The system may automatically blacklist these areas to not trigger a fall alert even if a "potential fall" is detected from the machine learning algorithm. For example, as discussed above, if a person is lying or sleeping in bed, the machine learning algorithm may detect a "possible fall". Since lying or sleeping in bed is a normal action, such an action may not trigger a fall alert for the user. However, if the person actually falls on the floor, the machine learning algorithm may detect a "potential fall". The system then determines whether the potential fall is within or outside the blacklist area. If the potential fall is within a blacklisted area, the system will not send an alert. If the potential fall is outside a blacklisted area, the system may determine that the fall is a "confirmed fall" and trigger (e.g., send) a fall alert to the user.

[0101] The posture detection system may include many commercial applications and benefits. For example, with respect to senior living communities, the system may provide predictive insights and prescriptions. In that regard, the system may provide flags or notifications to caregivers for early intervention. In particular, the system may measure and track frailty, for example, based on analyzing baseline mobility patterns and changes in frailty. The system may also detect and / or flag abnormal activity (e.g., excessive time in bed or bathroom). The system may also detect and / or flag nighttime trips to the bathroom.

[0102] Prior systems typically require active walking and the use of wearables or the completion of questionnaires. However, the current system may offer the value proposition of being private and non-intrusive. The system may also passively sense actual behavior, regardless of behavioral changes.

[0103] Frailty is currently measured in a clinic or doctor's office. A doctor may administer a test to a patient that involves a series of activities that may take 5-10 minutes to complete. However, such a short test does not give a holistic view of the patient, and the test is in an artificial setting where the patient may be more focused, trying harder, etc. In that respect, frailty scores may differ over time due to different settings and efforts by the patient. The current system improves upon prior art frailty tests by using "longitudinal" tracking, which involves tracking a person's movements over time. Thus, the current system provides a more holistic view of frailty over time.

[0104] Using posture detection functionality, the system may provide notifications or reports about events (e.g., falls, abnormal behavior, etc.) to, for example, caregivers, building management systems, alarm systems, notification systems, and / or emergency response systems. The reports may include data associated with the event (e.g., fall), such as, for example, location, time, actions before and / or after the fall, nearby items (e.g., furniture), items held by the person (e.g., groceries, walker, cane, another person), etc. Compared to radar or other sensor devices, the system may be less expensive, deployed more quickly, and more easily deployed. The system may also analyze images to facilitate audits and / or compliance. For example, the system may use images over time to monitor or audit care provided (e.g., bed check was completed at 11pm last night). The system may also use images over time to measure time spent providing care (e.g., average minutes in the bathroom with the patient each day). The system may also be integrated with other systems, such as scheduling and reporting systems. For example, the system may obtain data about when an employee started a shift, when an employee ended a shift, the employee's name and / or identifier, the units of time claimed by the employee caring for a patient, etc. The system may compare such submitted data to actual data obtained from images of resident activity and / or resident care to determine the accuracy of such submitted data.

[0105] The detailed description of the various embodiments herein refers to the accompanying drawings and photographs, which show various embodiments by way of example. Although these various embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure, it should be understood that other embodiments may be realized and that logical and mechanical changes may be made without departing from the spirit and scope of the present disclosure. Thus, the detailed description herein is presented for illustrative purposes only, and not for limiting purposes. For example, the steps recited in any of the method or process descriptions may be performed in any order and are not limited to the order presented. Also, any of the functions or steps may be outsourced to or performed by one or more third parties. Modifications, additions, or omissions may be made to the systems, devices, and methods described herein without departing from the scope of the present disclosure. For example, the components of the systems and devices may be integrated or separated. Also, the operations of the systems and devices disclosed herein may be performed by more, fewer, or other components, and the methods described may include more, fewer, or other steps. In addition, the steps may be performed in any suitable order. As used herein, "each" refers to each member of a set or each member of a subset of a set. Additionally, any reference to the singular includes plural embodiments, and any reference to more than one component may include singular embodiments. Although specific advantages are enumerated herein, various embodiments may include some, none, or all of the enumerated advantages. Systems and methods are provided.

[0106] In the detailed description of this specification, references to "various embodiments," "one embodiment," "an embodiment," "an example embodiment," and the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but all embodiments may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is contemplated that implementing such feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described, is within the knowledge of a person skilled in the art. After perusal of the description, it will be apparent to a person skilled in the art how to implement the present disclosure in alternative embodiments.

[0107] Benefits, other advantages, and solutions to problems are described herein with respect to specific embodiments. However, benefits, advantages, solutions to problems, and any elements that may cause or make any benefit, advantage, or solution more prominent are not to be construed as important, required, or essential features or elements of the invention. The scope of the invention is accordingly not limited except by the appended claims, and reference to an element in the singular is not intended to mean "one and only," unless expressly so stated, but rather, is intended to mean "one or more." Also, when a phrase similar to "at least one of A, B, or C" is used in the claims, it is intended that the phrase be interpreted to mean that only A may be present in an embodiment, only B may be present in an embodiment, only C may be present in an embodiment, or any combination of elements A, B, and C may be present in a single embodiment, i.e., for example, A and B, A and C, B and C, or A and B and C. Furthermore, no element, component, or method step in this disclosure is intended to be made available to the public, regardless of whether the element, component, or method step is expressly recited in a claim. No claim element herein is to be construed under the provisions of 35 U.S.C. 112(f) unless the element is expressly recited using the phrase "means for." As used herein, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements may include other elements not expressly recited or inherent to such process, method, article, or apparatus, rather than including only those elements.

[0108] Computer programs (also referred to as computer control logic) are stored in the main memory and / or the secondary memory. Computer programs may also be received via a communications interface. Such computer programs, when executed, enable the computer system to perform features as discussed herein. In particular, the computer programs, when executed, enable the processor to perform features of the various embodiments. Thus, such computer programs represent controllers of the computer system.

[0109] These computer program instructions may be loaded onto a general purpose computer, special purpose computer, or other programmable data processing apparatus and executed on the computer or other programmable data processing apparatus to produce a machine such that the instructions create means for implementing the functions defined in the flowchart block or blocks. These computer program instructions may also be stored in a computer readable memory such that the instructions stored in the computer readable memory may direct the computer or other programmable data processing apparatus to function in a particular manner to produce an article of manufacture including instruction means that implement the functions defined in the flowchart block or blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus such that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions defined in the flowchart block or blocks, causing a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process.

[0110] In various embodiments, the software may be stored in a computer program product and loaded into a computer system using a removable storage drive, a hard disk drive, or a communications interface. The control logic (software), when executed by a processor, causes the processor to perform the functions of the various embodiments as described herein. In various embodiments, the hardware components may take the form of application specific integrated circuits (ASICs). Implementation of such hardware to perform the functions described herein will be apparent to one skilled in the art.

[0111] As will be appreciated by those skilled in the art, the system may be embodied as a customization of an existing system, an add-on product, a processing device running upgraded software, a standalone system, a distributed system, a method, a data processing system, a device for data processing, and / or a computer program product. Thus, any part of the system or modules may take the form of a processing device running code, an Internet-based embodiment, an entirely hardware embodiment, or an embodiment combining Internet, software, and hardware aspects. Furthermore, the system may take the form of a computer program product on a computer-readable storage medium having computer-readable program code means embodied in the storage medium. Hard disk, CD-ROM, BLU-RAY® DISC (登録商標) Any suitable computer readable storage medium may be utilized including, optical storage devices, magnetic storage devices, and / or the like.

[0112] In various embodiments, the components, modules, and / or engines of the system may be implemented as micro-applications or micro-apps. Micro-apps are typically implemented in the mobile operating systems, such as the WINDOWS mobile operating systems, ANDROID (登録商標) Operating System, APPLE (登録商標)iOS Operating System, BLACKBERRY (登録商標) The micro-apps are deployed in the context of a mobile operating system, including Microsoft Corporation's operating systems, and equivalents. The micro-apps may be configured to leverage the resources of the larger operating system and associated hardware through a set of predefined rules that govern the operation of the various operating system and hardware resources. For example, if the micro-app desires to communicate with a device or network other than the mobile device or mobile operating system, the micro-app may leverage the communications protocols of the operating system and associated device hardware under the predefined rules of the mobile operating system. Also, if the micro-app desires input from a user, the micro-app may be configured to monitor various hardware components and then request a response from the operating system that communicates the detected input from the hardware to the micro-app.

[0113] The present system and method may be described herein in terms of functional block components, screenshots, options, and various processing steps. It should be understood that such functional blocks may be realized by any number of hardware and / or software components configured to perform the specified functions. For example, the present system may employ various integrated circuit components, such as memory elements, processing elements, logic elements, look-up tables, and the like, that may perform various functions under the control of one or more microprocessors or other control devices. Similarly, the software elements of the present system may be implemented using any combination of programming languages ​​such as C, C++, C#, JAVA, JAVASCRIPT, JAVASCRIPT Object Notation (JSON), VBScript, Macromedia COLD FUSION, COBOL, MICROSOFT C#, JAVA ... (登録商標) Active Server Pages, assembly, PERL (登録商標) , PHP, awk, PYTHON (登録商標) , Visual Basic, SQL Stored Procedures, PL / SQL, Any UNIX (登録商標) The system may be implemented using any programming or scripting language, such as shell script, and Extensible Markup Language (XML). Additionally, it should be noted that the system may employ any number of conventional techniques for data transmission, signaling, data processing, network control, and the like. Still further, the system may be used to detect or prevent security issues with client-side scripting languages, such as JAVASCRIPT, VBScript, or the like.

[0114] The present systems and methods are described herein with reference to screenshots, block diagrams, and flowchart illustrations of methods, apparatus, and computer program products according to various embodiments. It will be understood that each functional block of the block diagrams and flowchart illustrations, and combinations of functional blocks in the block diagrams and flowchart illustrations, respectively, can be implemented by computer program instructions.

[0115] Thus, the functional blocks in the block diagrams and flowchart illustrations support combinations of means for performing the specified functions, combinations of steps for performing the specified functions, and program instruction means for performing the specified functions. It should also be understood that each functional block in the block diagrams and flowchart illustrations and combinations of functional blocks in the block diagrams and flowchart illustrations can be implemented by either a special purpose hardware-based computer system or a suitable combination of special purpose hardware and computer instructions that perform the specified functions or steps. Furthermore, the illustrations of the process flow and their description may refer to a user's WINDOWS® application, web pages, websites, web forms, prompts, and the like. Practitioners should be aware that the illustrated steps described herein may be implemented by a WINDOWS® application, web pages, websites, web forms, prompts, and the like. (登録商標) APPLICATIONS, WEB PAGES, WEB FORMS, POPUP WINDOWS (登録商標) It will be appreciated that the steps may be arranged in any number of configurations, including the use of applications, prompts, and the like. Additionally, multiple steps as shown and described may be implemented using a single web page and / or WINDOWS (登録商標) It should be understood that the steps shown and described as single process steps may be combined into a single web page and / or WINDOWS application, but have been expanded for simplicity. In other cases, steps shown and described as single process steps may be combined into a single web page and / or WINDOWS application, but have been expanded for simplicity. (登録商標) Although they could be separated into applications, they have been combined for simplicity.

[0116] The middleware may include any hardware and / or software suitably configured to facilitate communication and / or process transactions between heterogeneous computing systems. Middleware components are commercially available and known in the art. The middleware may be implemented through commercially available hardware and / or software, through custom hardware and / or software components, or through a combination thereof. The middleware may reside in a variety of configurations, may exist as a stand-alone system, or may be a software component resident on an Internet server. The middleware may be configured to process transactions between various components of an application server and any number of internal or external systems for any of the purposes disclosed herein. IBM (登録商標) WEBSPHERE by , Inc. (Armonk, NY) (登録商標) MQTM (formerly MQSeries) is an example of a commercially available middleware product. An Enterprise Service Bus ("ESB") application is another example of middleware.

[0117] The computers discussed herein may provide a suitable website or other internet-based graphical user interface that is accessible by a user. (登録商標) Internet Information Services (IIS), Transaction Server (MTS) Services, and SQL SERVER (登録商標) The database is Microsoft (登録商標) Operating system: WINDOWS NT (登録商標) Web server software, SQL SERVER (登録商標) Databases, and Microsoft (登録商標) Used in conjunction with the commerce server. In addition, ACCESS (登録商標) Software, SQL Server (登録商標)Database, ORACLE (登録商標) Software, SYBASE (登録商標) Software, INFORMIX (登録商標) Software, MYSQL (登録商標) Software, INTERBASE (登録商標) Components such as software may be used to provide an Active Data Objects (ADO) compliant database management system. (登録商標) The web server is Linux (登録商標) Operating System, MYSQL (登録商標) Databases, and PERL (登録商標) , PHP, Ruby, and / or PYTHON (登録商標) It is used in conjunction with a programming language.

[0118] For the sake of brevity, conventional data networking, application development, and other functional aspects of the system (and components of the individual operating components of the system) may not be described in detail herein. Furthermore, the connecting lines shown in the various figures contained herein are intended to represent example functional relationships and / or physical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in a practical system.

[0119] In various embodiments, the methods described herein are implemented using various specific machines described herein. The methods described herein may be implemented using the specific machines described below and those developed in the future, in any suitable combination, as would be readily understood by one of ordinary skill in the art. Furthermore, as will be apparent from this disclosure, the methods described herein may result in various modifications of an article.

[0120] In various embodiments, the system and various components may be integrated with one or more smart digital assistant technologies. For example, an exemplary smart digital assistant technology is Amazon(登録商標) ALEXA developed by (登録商標) GOOGLE HOME system developed by Alphabet, Inc. (登録商標) System, Apple (登録商標) HOMEPOD (登録商標) systems, and / or similar digital assistant technologies. (登録商標) System, GOOGLE HOME (登録商標) System, and HOMEPOD (登録商標) Each system can provide cloud-based voice-activated services that can assist with tasks, entertainment, general information, and more. (登録商標) , AMAZON ECHO DOT (登録商標) , AMAZON TAP (登録商標) , and AMAZON FIRE (登録商標) All ALEXA TVs etc. (登録商標) The device is ALEXA (登録商標) Has access to the system. ALEXA (登録商標) System, GOOGLE HOME (登録商標) System, and HOMEPOD (登録商標) The system may receive voice commands, activate other functions, control smart devices, and / or gather information through its voice activation technology. For example, smart digital assistant technology may be used to interact with music, email, text, phone calls, answer questions, answer home improvement information, smart home communication / activation, play games, shop, create to-do lists, set alarms, stream podcasts, play audio books, and provide other real-time information such as weather, traffic, and news. ALEXA (登録商標) , GOOGLE HOME (登録商標) , and HOMEPOD (登録商標) The system may also allow users to access information about eligible transaction accounts linked to their online accounts across all digital assistant-enabled devices.

[0121] The various system components discussed herein may include one or more of the following: a host server or other computing system including a processor for processing digital data; a memory coupled to the processor for storing digital data; an input digitizer coupled to the processor for inputting digital data; an application program stored in the memory and accessible by the processor to direct the processing of the digital data by the processor; a display device coupled to the processor and memory for displaying information derived from the digital data processed by the processor; and a number of databases. The various databases used herein may include client data, merchant data, financial institution data, and / or similar data useful in the operation of the present system. As one skilled in the art would appreciate, a user computer may include an operating system (e.g., WINDOWS (登録商標) , UNIX (登録商標) , LINUX (登録商標) , SOLARIS (登録商標) , MAC OS (登録商標) etc.), as well as various conventional support software and drivers typically associated with a computer.

[0122] The system or any portion or function thereof may be implemented using hardware, software, or a combination thereof, and may be implemented in one or more computer systems or other processing systems. However, operations performed by the embodiments may be referred to in terms such as matching or selecting, which are generally associated with intellectual operations performed by a human operator. No such capability of a human operator is necessary or desirable in most cases in any of the operations described herein. Rather, the operations may be machine operations, or any of the operations may be performed or enhanced by artificial intelligence (AI) or machine learning. AI may generally refer to the study of agents (e.g., machines, computer-based systems, etc.) that perceive the world around them, form plans, and make decisions to achieve their goals. The foundations of AI include mathematics, logic, philosophy, probability, linguistics, neuroscience, and decision theory. Many fields, such as computer vision, robotics, machine learning, and natural language processing, fall under the umbrella of AI. Useful machines for implementing various embodiments include general purpose digital computers or similar devices.

[0123] In various embodiments, the embodiments are directed to one or more computer systems capable of performing the functionality described herein. The computer system includes one or more processors. The processors are connected to a communication infrastructure (e.g., a communication bus, a crossover bar, a network, etc.). Various software embodiments are described in terms of this exemplary computer system. After perusing this description, it will be apparent to one of ordinary skill in the art how to implement various embodiments using other computer systems and / or architectures. The computer system may include a display interface that transfers graphics, text, and other data from the communication infrastructure (or from a frame buffer, not shown) for display on a display unit.

[0124] The computer system also includes a main memory, such as random access memory (RAM), and may also include a secondary memory. The secondary memory may include, for example, a hard disk drive, a solid state drive, and / or a removable storage drive. The removable storage drive reads from and / or writes to a removable storage unit in a well-known manner. As will be appreciated, the removable storage unit includes a computer usable storage medium having computer software and / or data stored therein.

[0125] In various embodiments, secondary memory may include other similar devices for allowing computer programs or other instructions to be loaded into a computer system. Such devices may include, for example, removable storage units and interfaces. Examples of such may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as erasable programmable read-only memories (EPROMs), programmable read-only memories (PROMs), etc.) and associated sockets, or other removable storage units and interfaces that allow software and data to be transferred from the removable storage units to the computer system.

[0126] The terms "computer program medium," "computer usable medium," and "computer readable medium" are generally used to refer to media installed in a hard disk drive, such as a removable storage drive and a hard disk. These computer program products provide software to a computer system.

[0127] The computer system may also include a communications interface. The communications interface allows software and data to be transferred between the computer system and external devices. Examples of such communications interfaces may include a modem, a network interface (such as an Ethernet card), a communications port, and the like. The software and data transferred through the communications interface are in the form of signals, which may be electronic, electromagnetic, optical, or other signals capable of being received by the communications interface. These signals are provided to the communications interface over a communications path (e.g., channel). The channels carry the signals and may be implemented using wires, cables, optical fibers, telephone lines, cellular links, radio frequency (RF) links, wireless, and other communications channels.

[0128] As used herein, an "identifier" may be any suitable identifier that uniquely identifies an item. For example, the identifier may be a globally unique identifier ("GUID"). The GUID may be an identifier created and / or implemented under the Universally Unique Identifier standard. The GUID may also be stored as a 128-bit value that may be displayed as 32 hexadecimal digits. The identifier may also include a major number and a minor number. The major number and the minor number may each be a 16-bit integer.

[0129] In various embodiments, the server may be an application server (e.g., a WEB SPHERE (登録商標) ,WEBLOGIC (登録商標) ,JBOSS (登録商標) , POSTGRES PLUS ADVANCED SERVER (登録商標) In various embodiments, the server may include a web server (e.g., Apache, IIS, GOOGLE, etc.). (登録商標) Web Server, SUN JAVA(R) System Web Server, LINUX (登録商標) or WINDOWS (登録商標)It may also include a JAVA Virtual Machine running on the operating system.

[0130] A web client includes any device or software that communicates over any network, such as, for example, any device or software discussed herein. A web client may include Internet browsing software that is installed within a computing unit or system to perform online transactions and / or communications. These computing units or systems may take the form of a computer or set of computers, although other types of computing units or systems may be used, including personal computers, laptops, notebooks, tablets, smartphones, cellular phones, personal digital assistants, servers, pooled servers, mainframe computers, distributed computing clusters, kiosks, terminals, point of sale (POS) devices or terminals, televisions, or any other device capable of receiving data over a network. A web client may be a computer that is running an operating system (e.g., WINDOWS (登録商標) , WINDOWS MOBILE (登録商標) Operating System, UNIX (登録商標) Operating system, LINUX (登録商標) Operating System, APPLE (登録商標) OS (登録商標) The Web client may also include a variety of conventional support software and drivers typically associated with a computer. (登録商標) INTERNET EXPLORER (登録商標) Software, MOZILLA (登録商標) FIREFOX (登録商標) Software, Google Chrome TM Software, APPLE (登録商標) SAFARI (登録商標)The user may launch the software, or any of a number of other software packages available for browsing the Internet.

[0131] As one skilled in the art would appreciate, a web client may or may not be in direct contact with a server (e.g., an application server as discussed herein, a web server, etc.). For example, a web client may access the services of a server through another server and / or hardware component, which may have a direct or indirect connection to an Internet server. For example, a web client may communicate with a server through a load balancer. In various embodiments, the web client access is through a network or the Internet through a commercially available web browser software package. In that regard, the web client may be in a home or business environment with access to a network or the Internet. The web client may implement security protocols such as Secure Sockets Layer (SSL) and Transport Layer Security (TLS). The web client may implement several application layer protocols, including HTTP, HTTPS, FTP, and SFTP.

[0132] The various system components may be used independently, separately, or collectively, for example, in a variety of applications, including standard modem communications, cable modems, DISH NETWORK (登録商標) The network may be suitably coupled to the network via a data link, including a connection to an Internet Service Provider (ISP), via ISDN, Digital Subscriber Line (DSL), or a local loop such as those used in connection with various wireless communication methods. It should be noted that the network may be implemented as other types of networks, such as an interactive television (ITV) network. The system also contemplates the use, sale, or distribution of any goods, services, or information over any network having similar functionality as described herein.

[0133] The system contemplates use in association with web services, utility computing, pervasive and personalized computing, security and identification solutions, autonomous computing, cloud computing, commodity computing, mobility and wireless solutions, open source, biometrics, grid computing, and / or mesh computing.

[0134] Any of the communication, input, storage, database, or display discussed herein may be facilitated through a website having a web page. The term "web page" as used herein is not meant to limit the types of documents and applications that may be used to interact with a user. For example, a typical website may include various forms of JAVA applets, JAVASCRIPT programs, Active Server Pages (ASP), Common Gateway Interface Script (CGI), Extensible Markup Language (XML), Dynamic HTML, Cascading Style Sheets (CSS), AJAX (Asynchronous JAVASCRIPT and XML) programs, helper applications, plug-ins, and the like, in addition to standard HTML documents. A server may include a web service that receives a request from a web server, the request including a URL and an IP address (192.168.1.1). The web server retrieves the appropriate web page and transmits the data or application for the web page to the IP address. A web service is an application that is capable of interacting with other applications via a communication means such as the Internet. Web services are typically based on standards or protocols such as XML, SOAP, AJAX, WSDL, and UDDI. Web service methods are well known in the art and are covered in many standard texts. For example, Representative State Transfer (REST) ​​or RESTful Web Services may provide one method for enabling interoperability between applications.

[0135] The web client's computing unit may further be equipped with an Internet browser, connected to the Internet or an intranet using standard dial-up, cable, DSL, or any other Internet protocol known in the art. Transactions occurring at the web client may pass through a firewall to prevent unauthorized access from users of other networks. Furthermore, additional firewalls may be deployed between various components of the CMS to further enhance security.

[0136] Encryption may be performed using any of the techniques currently available or that may become available in the art, such as Twofish, RSA, El Gamal, Schorr signatures, DSA, PGP, PKI, GPG (GnuPG), HPE Format Preserving Encryption (FPE), Voltage, Triple DES, Blowfish, AES, MD5, HMAC, IDEA, RC6, and symmetric and asymmetric cryptosystems. The systems and methods may also incorporate SHA-series cryptography, Elliptic Curve Cryptography (e.g., ECC, ECDH, ECDSA, etc.), and / or other post-quantum cryptography algorithms in development.

[0137] The firewall may include any hardware and / or software suitably configured to protect the CMS components and / or enterprise computing resources from users of other networks. Additionally, the firewall may be configured to limit or restrict access to various systems and components behind the firewall for web clients connecting through the web server. The firewall may reside in a variety of configurations including stateful inspection, proxy-based, access control lists, and packet filtering, among others. The firewall may be integrated within the web server or any other CMS component, or may even reside as a separate entity. The firewall may implement network address translation ("NAT") and / or network address port translation ("NAPT"). The firewall may accommodate various tunneling protocols to facilitate secure communications such as those used in virtual private networking. The firewall may implement a demilitarized zone ("DMZ") to facilitate communications with public networks such as the Internet. The firewall may take the form of a software, integrated within the Internet server or any other application server component, resident within a separate computing device, or a stand-alone hardware component.

[0138] Any database discussed herein may include relational, hierarchical, graphical, block chain, object-oriented structures, and / or any other database configuration. Any database may also include a flat file structure in which data may be stored in a single file in the form of rows and columns without any structure for indexing and without any structural relationships between records. For example, the flat file structure may include a delimited text file, a CSV (comma separated values) file, and / or any other suitable flat file structure. Common database products that may be used to implement the database include IBM (登録商標) DB2 by Armonk, NY (登録商標) , ORACLE (登録商標) various database products available from Microsoft Corporation (Redwood Shores, CA); (登録商標) MICROSOFT ACCESS by Microsoft Corporation (Redmond, Washington) (登録商標) or MICROSOFT SQL SERVER (登録商標) MYSQL by MYSQL AB (Uppsala, Sweden) (登録商標) , MONGODB (登録商標) , DYNAMO DB (登録商標) Redis, Apache Cassandra (登録商標) , APACHE (登録商標) By HBASE (登録商標) , MAPR (登録商標) Examples of suitable database products include MapR-DB by Microsoft Corporation, or any other suitable database product. Also, any database may be organized in any suitable manner, for example, as a data table or a lookup table. Each record may be a single file, a series of files, a series of linked data fields, or any other data structure.

[0139] As used herein, big data may refer to partially or fully structured, semi-structured, or unstructured datasets that contain millions of rows and hundreds of thousands of columns. Big datasets may be compiled, for example, from purchase transaction history over time, from web registrations, from social media, from records of claims (ROC), from summaries of claims (SOC), from internal data, or from other suitable sources. Big datasets may be compiled without descriptive metadata such as column types, counts, percentiles, or other interpretation-aiding data points.

[0140] Association of certain data may be accomplished through any desired data association technique, such as those known or practiced in the art. For example, association may be accomplished either manually or automatically. Automatic association techniques may include, for example, database lookup, database merge, GREP, AGREP, SQL, using key fields in tables for fast lookup, sequential search through all tables and files, sorting records in files according to a known order to simplify lookup, and / or the like. Association steps may be accomplished by a database merge function, for example, using "key fields" in preselected databases or data sectors. Various database tuning steps are envisioned to optimize database performance. For example, frequently used files such as indexes may be placed on a separate file system to reduce input / output ("I / O") bottlenecks.

[0141] More specifically, a "key field" partitions the database according to the high-level class of objects defined by the key field. For example, a type of data may be designated as a key field in multiple related data tables, and the data tables may then be linked based on the type of data in the key field. The data corresponding to the key field in each of the linked data tables is preferably the same or of the same type. However, data tables having similar, but not identical, data in the key field may also be linked, for example, by using AGREP. According to an embodiment, any suitable data storage technique may be utilized to store the data without a standard format. The datasets may be stored using any suitable technique, including, for example, storing individual files using the ISO / IEC 7816-4 file structure, dedicated files are selected that implement the domain thereby exposing one or more base files containing one or more datasets, using datasets stored in individual files using a hierarchical filing system, datasets stored as records within a single file (including compressed, SQL accessible, hashed via one or more keys, numeric, alphabetic with first tuples, etc.), data stored as binary large objects (BLOBs), data stored as ungrouped data elements encoded using ISO / IEC 7816-6 data elements, data stored as ungrouped data elements encoded using ISO / IEC Abstract Syntax Notation (ASN.1) such as in ISO / IEC 8824 and 8825, fractal compression methods, image compression methods, and other proprietary techniques, which may include.

[0142] In various embodiments, the ability to store a wide variety of information in different formats is facilitated by storing the information as a BLOB. Thus, any binary information can be stored within the storage space associated with a dataset. As discussed above, the binary information may be stored in association with the system or external to but coordinated with the system. The BLOB method may store datasets as ungrouped data elements formatted as blocks of binary via fixed memory offsets using either fixed storage allocation, circular queue techniques, or best practices for memory management (e.g., least recently used paged memory, etc.). By using the BLOB method, the ability to store various datasets having different formats facilitates storage of data in a database or associated with the system by multiple unrelated owners of the datasets. For example, a first dataset that may be stored may be provided by a first party, a second dataset that may be stored may be provided by an unrelated second party, and a third dataset that may be stored may be provided by a third party unrelated to the first and second parties. Each of these three exemplary data sets may contain different information that is stored using different data storage formats and / or techniques. Additionally, each data set may contain subsets of data that may be distinct from the other subsets.

[0143] As described above, in various embodiments, data may be stored without regard to a common format. However, data sets (e.g., BLOBs) may be annotated in a standard manner when provided for manipulating data in a database or system. The annotations may comprise a short header, trailer, or other suitable indicator associated with each data set configured to convey information useful in managing various data sets. For example, the annotations may be referred to herein as "condition headers," "headers," "trailers," or "status," and may comprise an indication of the status of the data set, or may include an identifier correlated to a specific publisher or owner of the data. In one example, the first three bytes of each data set BLOB may be configured or configurable to indicate the status of that particular data set, e.g., LOADED, INITIALIZED, READY, BLOCKED, REMOVABLE, or DELETED. Subsequent bytes of data may be used to indicate, for example, an identification of the publisher, user, transaction / membership account identifier, or the like. Each of these condition annotations is discussed further herein.

[0144] Dataset annotations may also be used for other types of status information and for various other purposes. For example, a dataset annotation may include security information that establishes access levels. The access levels may be configured, for example, to allow only certain individuals, levels of employees, companies, or other entities to access the dataset, or to allow access to specific datasets based on transactions, merchants, issuers, users, or the like. Additionally, security information may restrict / allow only certain actions, such as accessing, modifying, and / or deleting the dataset. In one embodiment, the dataset annotation indicates that only the dataset owner or user is allowed to delete the dataset, and various identified users may be allowed to access the dataset for reading, while others are completely excluded from accessing the dataset. However, other access restriction parameters may also be used to allow various entities to access the dataset with various permission levels, as appropriate.

[0145] The data, including the header or trailer, may be received by a stand-alone interaction device configured to add, delete, modify, or extend the data in accordance with the header or trailer. Thus, in one embodiment, the header or trailer is not stored on the transaction device along with the associated issuer proprietary data, and instead appropriate action may be taken at the stand-alone device by providing the user with appropriate options regarding the action to be taken. The system may envision a data storage arrangement in which the header or trailer or header or trailer history of the data is stored on the system, device, or transaction instrument in association with the appropriate data.

[0146] Those skilled in the art will also understand that for security reasons, any database, system, device, server, or other component of the system may consist of any combination thereof, in a single location or multiple locations, and each database or system may include any of a variety of suitable security features, such as firewalls, access codes, encryption, decryption, compression, decompression, and / or the like.

[0147] Practitioners will also appreciate that there are several ways to display data within a browser-based document. Data may be represented as standard text or within fixed lists, scrollable lists, drop-down lists, editable text fields, fixed text fields, pop-up windows, and the like. Similarly, there are several ways available to modify data within a web page, such as, for example, free text entry using a keyboard, selection of menu items, check boxes, option boxes, and the like.

[0148] The data may be big data that is processed by a distributed computing cluster, such as a HADOOP cluster configured to process and store big data sets with some of the nodes having a distributed storage system and some of the nodes having a distributed processing system. (登録商標) In that regard, a distributed computing cluster may be a HADOOP software cluster, as defined by the Apache Software Foundation at www.hadoop.apache.org / docs. (登録商標) It may be configured to support a software distributed file system (HDFS).

[0149] As used herein, the term "network" includes any cloud, cloud computing system, or electronic communication system or method incorporating hardware and / or software components. Communication between parties may occur over, for example, a telephone network, an extranet, an intranet, the Internet, interaction point devices (point of sale devices, personal digital assistants (e.g., IPHONE (登録商標) Device, BLACKBERRY (登録商標) The communication may be accomplished through any suitable communication channel, such as a mobile telephone, a cellular phone, a kiosk, etc., online communication, satellite communication, offline communication, wireless communication, transponder communication, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), networked or linked devices, a keyboard, a mouse, and / or any suitable communication or data entry modality. Also, although the system is often described herein as being implemented using TCP / IP communication protocols, the system may also be implemented using protocols such as IPX, APPLETALK, etc. (登録商標) The network may be implemented using protocols such as programs, IP-6, NetBIOS, OSI, any tunneling protocol (e.g., IPsec, SSH, etc.), or any number of existing or future protocols. Where the network is in the nature of a public network such as the Internet, it may be advantageous to assume that the network is unsecure and exposed to eavesdroppers. Specific information relating to protocols, standards, and application software utilized in connection with the Internet is generally known to those skilled in the art and, therefore, need not be detailed herein.

[0150] "Cloud" or "cloud computing" includes a model for enabling opportunistic, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. Cloud computing may include location-independent computing whereby shared servers provide resources, software, and data to computers and other devices on demand.

[0151] As used herein, "transmitting" may include sending electronic data from one system component to another via a network connection. Additionally, as used herein, "data" may include generic information such as commands, queries, files, data for storage, and the like in digital or any other form.

[0152] Any database discussed herein may comprise a distributed ledger maintained by multiple computing devices (e.g., nodes) via a peer-to-peer network. Each computing device maintains a copy and / or a partial copy of the distributed ledger and communicates with one or more other computing devices in the network to verify and write data to the distributed ledger. The distributed ledger may use the features and functionality of blockchain technology, including, for example, common-law based verification of data, immutability, and cryptographically chained blocks. The blockchain may comprise a ledger of interconnected blocks that contain data. The blockchain may provide improved security because each block may hold the results of individual transactions and any blockchain executables. Each block may link to a previous block and may include a timestamp. Blocks may be linked because each block may include a hash of a prior block in the blockchain. The linked blocks form a chain, and only one subsequent block is allowed to link to one other predecessor block for a single chain. Although forks may be possible if divergent chains are established from a previously homogenous blockchain, typically only one of the divergent chains will be maintained as a shared chain. In various embodiments, the blockchain may implement smart contracts that enforce data workflow in a non-monopoly manner. The system may also include applications deployed on user devices, such as computers, tablets, smartphones, Internet of Things devices ("IoT" devices), etc. The applications may communicate with the blockchain (e.g., directly or via blockchain nodes) to transmit and read data. In various embodiments, a governing organization or entity may control access to data stored on the blockchain. Registration with a governing organization may allow for involvement within the blockchain network.

[0153] Data transfers performed through a blockchain-based system may propagate to peers connected within the blockchain network within a duration that may be determined by the block creation time of the specific blockchain technology implemented. For example, ETHEREUM (登録商標) On a HYPERLEDGER-based network, new data entries can be available within about 13-20 seconds of being written. (登録商標) On a Fabric 1.0-based platform, duration is driven by a concrete shared term algorithm that can be selected and implemented within seconds. In that respect, propagation times in the present system can be improved compared to existing systems, and implementation costs and time to market can also be significantly reduced. The present system also provides increased security, at least in part due to the immutable nature of the data stored in the blockchain, reducing the probability of tampering with various data inputs and outputs. The present system can also provide increased security of data by performing cryptographic processes on the data prior to storing the data on the blockchain. Thus, by transmitting, storing, and accessing data using the system described herein, the security of the data is improved, which reduces the risk of a computer or network being compromised.

[0154] In various embodiments, the system may also reduce database synchronization errors by providing a common data structure, thus improving, at least in part, the integrity of stored data. The system also provides increased reliability and fault tolerance over traditional databases (e.g., relational databases, distributed databases, etc.) because each node operates with a full copy of stored data, thus reducing downtime due, at least in part, to localized network and hardware failures. The system may also increase the reliability of data transfer in a network environment having reliable and unreliable peers, because each node broadcasts messages to all connected peers, and because each block includes a link to the previous block, nodes may quickly detect missing blocks and propagate requests for missing blocks to other nodes in the blockchain network.

[0155] The particular blockchain implementation described herein provides improvements over conventional techniques by using a decentralized database and an improved processing environment. In particular, blockchain implementations improve computer performance, for example, by leveraging decentralized resources (e.g., lower latency). Distributed computing resources improve computer performance, for example, by reducing processing time. Additionally, distributed computing resources improve computer performance, for example, by improving security using cryptographic protocols.

[0156] Any communication, transmission, and / or channel discussed herein may include any system or method for delivering content (e.g., data, information, metadata, etc.) and / or the content itself. The content may be presented in any form or medium, and in various embodiments, the content may be electronically delivered and / or capable of being presented. For example, a channel may be a website, a mobile application, or a device (e.g., Facebook,(登録商標) , YOUTUBE (登録商標) , PANDORA (登録商標) , APPLE TV (登録商標) , Microsoft (登録商標) XBOX (登録商標) , ROKU (登録商標) , AMAZON FIRE (登録商標) , GOOGLE CHROMECAST TM , SONY (登録商標) PLAYSTATION (登録商標) , NINTENDO (登録商標) SWITCH (登録商標) etc.), Uniform Resource Locators ("URLs"), Documents (e.g., Microsoft (登録商標) WORD or EXCEL TM , ADOBE (登録商標) Portable Document Format (PDF) documents, "eBooks," "emagazines," applications or micro-applications (as described herein), Short Message Service (SMS) or other types of text messages, email, Facebook, (登録商標) MESSAGE, TWITTER (登録商標) The channels may comprise tweets, multimedia messaging services (MMS), and / or other types of communication technologies. In various embodiments, the channels may be hosted or provided by a data partner. In various embodiments, the distribution channels may comprise at least one of a merchant website, a social media website, an affiliate or partner website, an external vendor, a mobile device communication, a social media network, and / or a location-based service. The distribution channels may include at least one of a merchant website, a social media site, an affiliate or partner website, an external vendor, and a mobile device communication. An example of a social media site is FACEBOOK. (登録商標) , FOURSQUARE (登録商標) , TWITTER (登録商標) , LINKEDIN (登録商標) , INSTAGRAM (登録商標), PINTEREST (登録商標) , TUMBLR (登録商標) ,REDDIT (登録商標) , SNAPCHAT (登録商標) , WHATSAPP (登録商標) , FLICKR (登録商標) , V.K. (登録商標) , QZONE (登録商標) , WECHAT (登録商標) Examples of affiliate or partner websites include AMERICAN EXPRESS, (登録商標) , GROUPON (登録商標) , LIVING SOCIAL (登録商標) etc. (Item 1) 1. A method comprising: receiving, by a processor, an image of a human from a sensor; receiving, by the processor, a location of a bounding box on the image, the bounding box containing pixel data of the human in the image; obtaining, by the processor, bounding box data from within the bounding box; determining, by the processor, a pose of the human based on the bounding box data; A method comprising: (Item 2) 2. The method of claim 1, further comprising training, by the processor, a neural network to predict the placement of the bounding box on the image. (Item 3) 2. The method of claim 1, further comprising training, by the processor, the neural network using the pixel data, the human thermal data, and environmental data. (Item 4) 2. The method of claim 1, further comprising adjusting, by the processor, an algorithm of the neural network based on the environmental data. (Item 5) 13. The method of claim 1, further comprising adjusting, by the processor, a neural network algorithm based on environmental data, the environmental data comprising at least one of an environmental temperature, an indoor temperature, a floor plan, a non-human thermal object, a gender of the human, an age of the human, a height of the sensor, clothing of the human, or a weight of the human. (Item 6) 2. The method of claim 1, wherein the sensor obtains thermal data about the human. (Item 7) 2. The method of claim 1, wherein a user indicates placement of the bounding box on the image. (Item 8) 2. The method of claim 1, wherein determining the pose is with respect to a frame of the image captured by the sensor. (Item 9) 2. The method of claim 1, wherein determining the pose further comprises determining an aggregate pose over a period of time across a plurality of frames. (Item 10) 2. The method of claim 1, further comprising determining, by the processor, a fall based on aggregate posture changes from at least one of a standing posture or a sitting posture to a lying posture and the lying posture sustained for an amount of time. (Item 11) 2. The method of claim 1, further comprising extracting, by the processor, differentiating features from the images in multiple frames using pattern recognition. (Item 12) 2. The method of claim 1, further comprising limiting, by the processor, a resolution of the image based on at least one of privacy concerns, power consumption of the sensor, cost of the pixel data, bandwidth associated with the pixel data, computation cost, or computation bandwidth. (Item 13) 2. The method of claim 1, further comprising labeling, by the processor, a pose of the person in the image. (Item 14) 2. The method of claim 1, wherein the image is part of a video footage of the person. (Item 15) 2. The method of claim 1, wherein obtaining the bounding box data includes obtaining the bounding box data at least one of over time or during an initial calibration session. (Item 16) 2. The method of claim 1, further comprising analyzing, by the processor, differentiating features of a pattern from an overhead thermal signature of the human to determine a posture of the human. (Item 17) 2. The method of claim 1, wherein the posture includes at least one of sitting, standing, lying, exercising, dancing, running, or eating. (Item 18) determining, by the processor, a temperature of the human within the space based on infrared (IR) energy data of IR energy from the human; determining, by the processor, location coordinates of the human within the space; comparing, by the sensor system, location coordinates of the human with location coordinates of a fixed object; determining, by the sensor system, that the human is a person in response to a temperature of the object being within a range and in response to location coordinates of the human being being distinct from location coordinates of the fixed object; The method of claim 1, further comprising: 2. The method of claim 1, further comprising determining, by the processor, a trajectory of the human based on changes in temperature in the pixel data, the temperature being projected onto a grid of pixels. (Item 19) 1. An article of manufacture, the article of manufacture including a non-transitory tangible computer readable storage medium having instructions stored thereon that, in response to execution by a computer-based system, cause the computer-based system to: receiving, by the processor, an image of a human from a sensor; receiving, by the processor, a location of a bounding box on the image, the bounding box containing pixel data of the human in the image; obtaining, by the processor, bounding box data from within the bounding box; determining, by the processor, a pose of the human based on the bounding box data; An article of manufacture that performs an operation including: (Item 20) 1. A system comprising: A processor; a tangible non-transitory memory configured to communicate with the processor, the tangible non-transitory memory having instructions stored thereon, the instructions being responsive to execution by the processor to cause the processor to: receiving, by the processor, an image of a human from a sensor; receiving, by the processor, a location of a bounding box on the image, the bounding box containing pixel data of the human in the image; obtaining, by the processor, bounding box data from within the bounding box; determining, by the processor, a pose of the human based on the bounding box data; and A system comprising:

Claims

1. 1. A method comprising: receiving, by a processor, an image of a human from a sensor; creating, by the processor, a bounding box on the image using a convolutional neural network (CNN), the CNN using gradient descent based training based on a hand-drawn bounding box around an image of the human, pixel data of the human in the image, thermal data, and environmental data around the human; adjusting, by the processor, the bounding box based on the environmental data surrounding the human; obtaining, by the processor, bounding box data from within the bounding box; determining, by the processor, using the CNN to determine differentiating features of the image within the bounding box based on the bounding box data, a filter, and an overhead heat signature of the human presence, the CNN using stochastic gradient descent to create a feature map of the differentiating features; determining, by the processor, one or more poses of the human based on the bounding box data and the differentiating features; determining, by the processor, an aggregated pose from the one or more poses over a period of time across a plurality of frames, the aggregated pose smoothing variations in the one or more poses over the period of time across the plurality of frames; determining, by the processor, that the event is a potential fall based on the aggregated posture change from at least one of a standing posture or a sitting posture to a lying posture and the lying posture persisting for an amount of time based on a time stamp; A method comprising:

2. The method of claim 1 , wherein the aggregated poses are the poses that appear most frequently in the set of one or more poses.

3. The method of claim 1 , further comprising: limiting, by the processor, a resolution of the image based on at least one of privacy concerns, sensor power consumption, data cost, data bandwidth, computation cost, or computation bandwidth.

4. The CNN determining the differentiating features comprises: generating, by said processor, a vector of features of said image; learning, by the processor, a 2D representation of the image using the CNN; flattening, by the processor, the image; complementing, by the processor, the image with a positional encoding; receiving, by the processor, a number of learned position embeddings for an encoder output using a transformer decoder; transmitting, by the processor, the learned position embedding of the encoder output to a shared feed-forward network (FFN), the FFN providing a box prediction by predicting at least one of a class and the detection of the bounding box or a "no object" class; The method of claim 1 , comprising:

5. 10. The method of claim 1, further comprising: adjusting, by the processor, the CNN algorithm based on environmental data, the environmental data comprising at least one of an environmental temperature, an indoor temperature, a floor plan, a non-human thermal object, a gender of the human, an age of the human, a height of the sensor, clothing of the human, or a weight of the human.

6. applying, by the processor, a CNN to an interior of the bounding box; predicting, by the processor, a size and location of the bounding box; generating, by the processor, a pose prediction for the image based on gradient descent based training of the CNN; The method of claim 1 further comprising:

7. The method of claim 1 , wherein determining that the event is a potential fall is based on a confidence score.

8. The method of claim 1 , wherein determining the one or more poses is with respect to frames of the image captured by the sensor.

9. occluding, by said processor, spatial locations where said event cannot reasonably occur; or filtering, by the processor, the events based on a threshold of confidence in detection of the events. The method of claim 1 , further comprising post-processing by at least one of:

10. determining, by the processor, that the potential fall is within a blacklist spatial location; transmitting, by the processor, a potential fall alert based on the potential fall being within a blacklist location; The method of claim 1 further comprising:

11. determining, by the processor, that the potential fall is outside a blacklist spatial location; sending, by the processor, a confirmed fall alert based on the potential fall being outside a blacklist location; The method of claim 1 further comprising:

12. 10. The method of claim 1, further comprising: limiting, by the processor, a resolution of the image based on at least one of privacy concerns, power consumption of the sensor, cost of the pixel data, bandwidth associated with the pixel data, computation cost, or computation bandwidth.

13. The method of claim 1 , further comprising labeling, by the processor, one or more poses of the human in the image.

14. The method of claim 1 , wherein the image is part of a video footage of the person.

15. The method of claim 1 , wherein obtaining the bounding box data comprises obtaining the bounding box data at least one of over time or during an initial calibration session.

16. The method of claim 1 , further comprising analyzing, by the processor, the differentiating features of a pattern from an overhead thermal signature of the human to determine one or more poses of the human.

17. The method of claim 1 , wherein the one or more positions include at least one of sitting, standing, lying down, exercising, dancing, running, or eating.

18. determining, by the processor, a temperature of the human within the space based on infrared (IR) energy data of IR energy from the human; determining, by the processor, location coordinates of the human within the space; comparing, by the sensor system, location coordinates of the human with location coordinates of a fixed object; determining, by the sensor system, that the human is a person in response to a temperature of the object being within a range and in response to location coordinates of the human being being distinct from location coordinates of the fixed object; The method of claim 1 further comprising:

19. 1. An article of manufacture, the article of manufacture including a non-transitory tangible computer readable storage medium having instructions stored thereon that, in response to execution by a computer-based system, cause the computer-based system to: receiving, by a processor, an image of a human from a sensor; creating, by the processor, a bounding box on the image using a convolutional neural network (CNN), the CNN using gradient descent based training based on a hand-drawn bounding box around an image of the human, pixel data of the human in the image, thermal data, and environmental data around the human; adjusting, by the processor, the bounding box based on the environmental data surrounding the human; obtaining, by the processor, bounding box data from within the bounding box; determining, by the processor, using the CNN to determine differentiating features of the image within the bounding box based on the bounding box data, a filter, and an overhead heat signature of the human presence, the CNN using stochastic gradient descent to create a feature map of the differentiating features; determining, by the processor, one or more poses of the human based on the bounding box data and the differentiating features; determining, by the processor, an aggregated pose from the one or more poses over a period of time across a plurality of frames, the aggregated pose smoothing variations in the one or more poses over the period of time across the plurality of frames; determining, by the processor, that the event is a potential fall based on a change in the aggregated posture from at least one of a standing posture or a sitting posture to a lying posture and the lying posture persisting for an amount of time based on a time stamp; An article of manufacture that performs an operation including:

20. 1. A system comprising: A processor; a tangible non-transitory memory configured to communicate with the processor, the tangible non-transitory memory having instructions stored thereon, the instructions being responsive to execution by the processor to cause the processor to: receiving, by the processor, an image of a human from a sensor; creating, by the processor, a bounding box on the image using a convolutional neural network (CNN), the CNN using gradient descent based training based on a hand-drawn bounding box around an image of the human, pixel data of the human in the image, thermal data, and environmental data around the human; adjusting, by the processor, the bounding box based on the environmental data surrounding the human; obtaining, by the processor, bounding box data from within the bounding box; determining, by the processor, using the CNN to determine differentiating features of the image within the bounding box based on the bounding box data, a filter, and an overhead heat signature of the human presence, the CNN using stochastic gradient descent to create a feature map of the differentiating features; determining, by the processor, one or more poses of the human based on the bounding box data and the differentiating features; determining, by the processor, an aggregated pose from the one or more poses over a period of time across a plurality of frames, the aggregated pose smoothing variations in the one or more poses over the period of time across the plurality of frames; determining, by the processor, that the event is a potential fall based on a change in the aggregated posture from at least one of a standing posture or a sitting posture to a lying posture and the lying posture persisting for an amount of time based on a time stamp; and A system comprising:

Images