Gesture detection using hot data

Abstract

The system may provide functionality for gesture detection by using a gesture detection algorithm on the reduced resolution image. The method may include: receiving an image of a human from a sensor; receiving a position of a bounding box on the image, where the bounding box contains pixel data of a human in the image; obtaining bounding box data from the bounding box; and determining a gesture of the human based on the bounding box data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Serial No. 17 / 708,493, filed on March 30, 2022, and entitled “POSE DETECTION USING THERMALDATA.” U.S. Serial No. 17 / 708,493 is a continuation-in-part of, and claims priority to, U.S. Serial No. 17 / 516,954, filed on November 2, 2021, and entitled “USER INTERFACE FOR DETERMINING LOCATION, TRAJECTORY AND BEHAVIOR.” U.S. Serial No. 17 / 516,954 is a continuation-in-part of U.S. Serial No. 17 / 232,551, filed on April 16, 2021, and entitled “THERMAL DATA ANALYSIS FOR DETERMINING LOCATION, TRAJECTORY AND BEHAVIOR.” U.S. Serial No. 17 / 232,551 is a continuation of U.S. Serial No. 17 / 178,784, filed on February 18, 2021, and entitled “MONITORING HUMAN LOCATION, TRAJECTORY AND BEHAVIOR USING THERMAL DATA” (also known as U.S. Patent No. 11,022,495, issued on June 1, 2021). U.S. Serial No. 17 / 178,784 claims priority to U.S. Provisional Serial No. 62 / 986,442, filed on March 6, 2020, and entitled “MULTI-WIRELESS-SENSOR SYSTEM, DEVICE, AND METHOD FOR MONITORING HUMAN LOCATION AND BEHAVIOR”. All of the above disclosures are hereby incorporated by reference in their entirety for all purposes. Technical Field

[0003] The present disclosure relates generally to gesture detection using thermal data, and more particularly to fall detection and other activity analysis using gesture information. Background Art

[0004] While some merchants may attempt to use basic machines to count the number of people entering and leaving a specific door leading to a store, this information is very limited for analyzing the movements of these people within the store. Merchants may be very interested in better understanding the movement, trajectory, and activities of customers within the merchant's store. For example, a merchant may be interested in understanding whether a certain display in a specific aisle in the store attracts more customers to that aisle. In addition, a merchant may be interested in understanding how many customers who walked along aisle #4 also walked along aisle #5 and how many customers who walked along aisle #4 skipped aisle #5 and continued to walk along aisle #6. This data can help merchants optimize their operations and maximize profits.

[0005] Merchants may also be interested in gaining a more complete understanding of general traffic patterns in their stores as they relate to time. To help merchants better allocate their own resources and optimize their business relationships with partner third parties, merchants may want to understand traffic patterns during the busiest times of the day, during the busiest days of the week, during specific months, and / or during specific years. In addition, merchants may want to obtain more information about spatial and / or temporal patterns of traffic and occupancy levels to help identify abnormal behavior and / or detect incidents in real time.

[0006] In addition, assisted living providers often want to obtain spatial and temporal movement data of tenants in order to analyze the well-being of residents and determine whether residents are eligible for independent living. For example, the provider may want to analyze the movement speed of tenants based on their indoor positions over time, calculate the total calories consumed based on the movement of the tenants, and / or monitor the body temperature of the tenants.

[0007] Detecting human gestures may also be helpful for fall detection and other activity analysis. However, gesture detection is often required in private homes or other private environments where privacy is a concern. In this regard, users typically prefer that any data collection or scanner does not obtain or store personal identity information. Thus, a non-invasive technology is needed to implement such fall detection analysis in homes. In this regard, high-resolution cameras or other technologies may not be desirable because such cameras may use key points to obtain and / or store facial data or other personal identity information. Key points may include a subset of points on a human skeleton, which requires detailed information about humans. It is very difficult and often impossible to identify key points in low-resolution images, so high-resolution technology is typically required to identify key points. In contrast, residential environments tend to use low-resolution data. An example of low-resolution data may be thermal data. Thus, there is a need to use thermal data in combination with an algorithm that can process the thermal data to provide gesture detection.

[0008] In addition to cameras, other solutions have been used for fall detection, such as watches (e.g., using accelerometers and gyroscopes), radar, or lidar. However, watch technology requires a human to actively wear the technology. In addition, the use of radar may lead to false alarms (e.g., the radar may be triggered by a pet). In addition, radar technology as part of the detection process is generally not effective in distinguishing stationary people. In addition, existing systems may require analysis of densely aggregated data points, which generally results in lower accuracy. Analysis of such dense points may be more expensive because additional computing power may be required to distinguish points in a dense point cloud. Summary of the invention

[0009] In various embodiments, a system may implement a method comprising: receiving, by a processor, an image of a human being from a sensor; receiving, by the processor, a placement of a bounding box on the image, wherein the bounding box contains pixel data of the human being in the image; obtaining, by the processor, bounding box data from within the bounding box; and determining, by the processor, a posture of the human being based on the bounding box data.

[0010] In various embodiments, the method may further include training, by the processor, a neural network to predict the location of a bounding box on an image. The method may further include training, by the processor, the neural network using pixel data of a human, thermal data of a human, and environmental data. The method may further include adjusting, by the processor, an algorithm of the neural network based on the environmental data, wherein the environmental data includes at least one of: ambient temperature, room temperature, floor plan, non-human thermal objects, sex of the human, age of the human, height of the sensor, clothing of the human, or body weight of the human.

[0011] In various embodiments, the sensor may acquire thermal data about the human being. A user may indicate a location of a bounding box on an image. Determining a posture may be performed for a frame of an image captured by the sensor. Determining a posture may also include determining an aggregated posture across multiple frames over a period of time. The image may be part of a video clip of the human being. Acquiring bounding box data may include acquiring bounding box data at least one of over time or during an initial calibration session. The posture may include at least one of: sitting, standing, lying, exercising, dancing, running, or eating.

[0012] In various embodiments, the method may also include determining, by the processor, a fall based on the aggregated posture changing from at least one of a standing posture or a sitting posture to a lying posture and the lying posture persisting for a certain amount of time. The method may include extracting, by the processor, distinguishing features from multiple frames of the image using pattern recognition. The method may include limiting, by the processor, the resolution of the image based on at least one of the following: privacy issues, power consumption of the sensor, cost of pixel data, bandwidth of pixel data, computational cost, or computational bandwidth. The method may include labeling, by the processor, the posture of humans in the image.

[0013] In various embodiments, the method may also include: determining, by a processor, a temperature of a human being in space based on infrared (IR) energy data regarding infrared (IR) energy from the human being; determining, by a processor, position coordinates of the human being in space; comparing, by a sensor system, the position coordinates of the human being with position coordinates of a fixture; and determining, by a sensor system, that the human is a human being in response to the temperature of the object being within a range and in response to the position coordinates of the human being being different from the position coordinates of the fixture. The method may include analyzing, by a processor, distinguishing features of a pattern of thermal signatures from the top of the human being's head to determine a posture of the human being. The method may include determining, by a processor, a trajectory of the human being based on changes in temperature in pixel data, wherein the temperature is projected onto a pixel grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The subject matter of the present disclosure is particularly pointed out and distinctly claimed in the concluding portion of the specification. However, a more complete understanding of the disclosure may be best obtained by referring to the detailed description and claims when considered in conjunction with the accompanying drawings.

[0015] Figure 1A is an exemplary schematic diagram of the main components of a sensor node as part of an overall system according to various embodiments.

[0016] Figure 1B is an exemplary schematic diagram of a gateway and a microprocessor as part of an overall system according to various embodiments.

[0017] Figure 2 is an exemplary data flow diagram according to various embodiments.

[0018] Figure 3 is an exemplary system architecture according to various implementations.

[0019] Figure 4A and Figure 4B is an exemplary user interface according to various implementations.

[0020] Figure 5 are exemplary building layouts according to various implementations.

[0021] Figure 6 is an exemplary building layout illustrating certain sensor nodes and the coverage area for each sensor node in accordance with various embodiments.

[0022] Figure 7 is an exemplary user interface illustrating an application's response to detecting a sensor node in a physical space after a user successfully logs into the sensor node, in accordance with various embodiments.

[0023] Fig. 8A and Figure 8B is an exemplary user interface according to various embodiments that prompts a user to upload to an application any specific files that may serve as supplemental visual information for a representation of their physical space.

[0024] Fig. 9 is an exemplary user interface that illustrates certain sensor nodes located in a space that may be labeled with names so that a user understands the context of each sensor node in their equivalent space, in accordance with various embodiments.

[0025] Fig.10 is an exemplary user interface illustrating the ability to set up and calibrate a sensor node in accordance with various embodiments.

[0026] FIG. 11A to FIG. 11C Exemplary thermal signature patterns are shown, including a standing posture, a sitting posture, and a lying posture, according to various embodiments.

[0027] Fig.12 An exemplary posture inference process is shown in accordance with various implementations.

[0028] Fig.13 An exemplary fall detection process is shown in accordance with various embodiments. DETAILED DESCRIPTION

[0029] In various embodiments, the system is configured to locate, track and / or analyze the activities of organisms in the environment. The system does not require the input of biometric data of the person. Although the present disclosure may discuss human activities, the present disclosure contemplates tracking any item such as, for example, an animal or any object that can provide infrared (IR) energy. Although the present disclosure may discuss indoor environments, the system may also track in outdoor environments (e.g., outdoor concert venues, outdoor amusement parks, etc.) or a mixture of outdoor and indoor environments.

[0030] like Figure 1A , Figure 1B and Figure 3As described in more detail in , in various embodiments, the system may include multiple sensor nodes 102, a gateway 135, a microprocessor 140, a computing module 350 (e.g., a cloud computing module), a database 360, and / or a user interface 370 (e.g., Figure 4A , Figure 4B , Figure 7 Figure 8 Fig. 9 and Fig.10 ). Each sensor node 102 may include an enclosure 105 , an antenna 110 , a sensor module 115 , a switch 120 , a light emitting diode (LED) 125 , and a power source 130 .

[0031] In various embodiments, the sensor module 115 can be any type of sensor, such as a thermopile sensor module. The thermopile sensor module 115 can include, for example, a Heimann GmbH sensor module or a Panasonic AMG8833. Each sensor module 115 can be housed in an enclosure 105. The sensor module 115 is configured to measure temperature from a distance by detecting IR energy from an object (e.g., a living being). If the living being has a higher temperature, the living being will emit more IR energy. The thermopile sensing element in the thermopile sensor module 115 can include a thermocouple on a silicon chip. The thermocouple absorbs IR energy and generates an output signal indicating the amount of IR energy. In this way, a higher temperature causes more IR energy to be absorbed by the thermocouple, resulting in a higher signal output.

[0032] In various embodiments, the sensor node 102 interface can be wireless to help reduce the labor and material costs associated with installation. In various embodiments, each sensor node in the sensor node 102 can obtain power from any power source 130. The power source 130 can power one or more sensor nodes 102. Each sensor node in the sensor node 102 can be powered by a battery alone. The battery can be low enough power to work for more than 2 years using a single battery (e.g., a 19wh battery). The battery 130 can include batteries from any manufacturer and / or PKCELL batteries, D-cell batteries, or any other battery type. The battery 130 can be housed in a battery holder (e.g., a Bulgin battery holder). The system can also measure the battery voltage of the battery 130 (e.g., a D-cell battery). The battery voltage can be measured using an analog-to-digital converter located on-board with an antenna 110 (e.g., a Midatronics Dusty PCB antenna). When the battery voltage measurement is obtained, the system can also add a timestamp to the battery voltage data.

[0033] The system can be expanded to a larger footprint by adding more sensor nodes 102 to the array of sensor nodes 102. In various embodiments, sensor nodes 102 can be added dynamically, where Figure 7 An exemplary user interface for adding sensors is described in . In particular, a user can add a sensor node 102 to an established network of sensor nodes 102 or remove a sensor node 102 from an established network of sensor nodes 102 at any time or any place. The sensor node 102 can be located anywhere, as long as the sensor node 102 location is within the bandwidth of the gateway 135. In this way, a large number of sensor nodes 102 can form a mesh network to communicate with the gateway 135. The sensor node 102 can communicate with the gateway 135 at about the same time. Each new sensor node 102 connected to the gateway 135 can further extend the boundaries of the mesh network, improve system stability, and improve system performance. In this way, a large number of sensor nodes 102 can form a mesh network to communicate with the gateway 135. The sensor node 102 can communicate with the gateway 135 at about the same time. Each new sensor node 102 connected to the gateway 135 can further extend the boundaries of the mesh network, improve system stability, and improve system performance. The sensor node 102 can be placed or installed on any part of a building or on any object. For example, the sensor nodes 102 may be mounted to the ceiling, sidewalls, or floor of the desired space using any fasteners known in the art. For a 2.5 meter high ceiling, the distance between the sensor nodes 102 may be spaced, for example, 4 meters apart.

[0034] In various embodiments, each sensor node 102 installed in any given space may have a unique number (e.g., a MAC address) assigned to the sensor node 102. The system uses the unique numbers to create a structured network with the numbered sensor nodes 102. Figure 6 As shown in , each sensor node can cover a different area and carry the unique number that can be accessed by the user both in the digital environment and also in the physical environment. In particular, the unique number is clearly printed on the enclosure 105 of the sensor node and is published on the user's screen when the user is installing the sensor node. In this way, the user can identify whether the location of their physical sensor node matches the digital representation in the electronic space they have created in the installation application.

[0035] like Figure 7 and Figure 8BAs described in , after a sensor node 102 is added and set up in the system, the sensor node 102 creates a profile. The user is prompted to either scan a QR code located on the sensor, thereby automatically registering a digital representation of the physical sensor, or manually enter the MAC address of the sensor in question. The sensor node 102 profile may include the height, mac address, relative position in space, surrounding objects, and / or contextual information of the sensor node 102. The system determines the extent of coverage of each sensor node based on the height of the sensor node 102 that the user enters as information in its profile. Some examples of contextual information may include the name of the room in which the sensor is located, the name of the sensor itself (if the user wishes to assign a name), and the number assigned to the sensor. The user can upload files about the surrounding environment, such as Fig. 8A . Such files may include, for example, PDF, JPG, PNG, 3DM, OBJ, FBX, STL and / or SKP. Surrounding information may include furniture within the sensor node 102 field, building layout around the sensor node 102 field, etc. The user may decide to add surrounding objects within the space. The sensor node may not register surrounding objects; however, the surrounding objects can provide a richer visual environment for the user's own personnel use. The sensor node 102 profile can be stored in the user's profile in the database 360.

[0036] The thermopile sensor module 115 can project the temperature of the object onto a grid. The grid can be an 8 pixel x 8 pixel grid, a 16 pixel x 16 pixel grid, or a 32 pixel x 32 pixel grid (64 pixels, 256 pixels, or 1024 pixels, respectively). The thermopile sensor module 115 can be tuned to detect a specific thermal spectrum, allowing detection of objects with a standard temperature (such as the human body). The average normal body temperature is generally recognized as 98.6°F (37°C). However, normal body temperature can have a wide range, from 97°F (36.1°C) to 99°F (37.2°C). Higher temperatures generally indicate infection or disease. The system can detect such temperature differences because the sensor module 115 can have an accuracy of 0.5°C. In the case of multiple human bodies in the same area, the thermopile sensor module 115 captures and processes each body as a different heat source. In particular, the system includes a calibration process built into the 3D front end (an exemplary user interface is in Fig.10 ) to avoid overlapping temperature readings from different bodies.

[0037] As part of the calibration process (an example user interface is shown in Fig.10As part of the calibration process, the user is asked through the interface of the application to exit the coverage of all sensor modules 115 so that the system can automatically adjust the sensitivity of the sensor modules 115. Starting from the maximum sensitivity, the system gradually reduces its sensitivity until high-frequency noise is no longer detected. The absolute elimination of noise allows the body to be detected as a different heat source, and subsequently, two heat sources such as the human body are detected as different and independent entities. For spatial overlap readings between the "field of view" of two sensor modules 115, during the calibration process, the system identifies the overlap area between the two sensor modules 115 and averages the common detections between the two sensor modules 115. In the event that overlap is detected between more than two sensor modules 115, the system will average the overlap of each pair in sequence. For example, for the overlap between sensor modules 115A, B and C, the system averages A and B first, and then continues to average the results of AB with C.

[0038] If during the calibration process (the example user interface is Fig.10 ), the system automatically generates a digital path between the installed sensor nodes, prompting the user to physically stand under each sensor node. When doing so, the sensor node 102 detects the user's movement and the detection becomes visible within the application. If successful, the user is prompted to follow the path and thus complete the calibration of each sensor node 102 of the network. If any sensor does not respond as mentioned above, the user is prompted to digitally manipulate the sensitivity of the sensor module 115 by sliding a digital bar and adjusting the sensitivity level of the sensor node in question accordingly. More specifically, during troubleshooting of the calibration process, the user is prompted to stand under the physical sensor, at each of the four corners of its field of view - one corner at a time. At each corner, which is named A, B, C and D by default in the application, the user will be called to stay as long as the system detects their presence and successfully prints it on the digital double of the sensor. Once the location is detected, the application asks the user which of the four possible corners they are trying to mark.

[0039] In various embodiments, the thermopile sensor module 115 can detect an array of temperature readings. In addition to detecting the temperature of a living being based on pixel values, the thermopile sensor module 115 can also obtain the temperature of the environment in which the sensor module 115 is located. The system uses local information and global information (both individually for each pixel and across all the pixels of the sensor modules 115 that form the network) to determine what the background temperature field is. The thermopile sensor module 115 can obtain independent temperature measurements of the sensor node 102 itself. The temperature of the sensor module 115 can be obtained using an onboard thermocouple. The system can use the temperature of the sensor node 102 and / or the temperature of the sensor module 115 to give an assessment of the temperature profile of the monitored space. The onboard thermocouple measurement itself measures the temperature of the space where the sensor is located. The system uses bilinear interpolation to estimate the temperature between the sensor nodes 102 and / or sensor modules 115 in the space to approximate the temperature distribution. Additionally, in various embodiments, the system may measure and capture the temperature of the environment multiple times throughout the day to reduce the adverse effects of maintaining a fixed background temperature field for threshold calculations, thereby increasing overall detection accuracy in real-world scenarios where the ambient temperature is dynamic.

[0040] In various embodiments, multiple sensor nodes 102 can provide information in real time to help generate real-time location, trajectory and / or behavior analysis of human activities. By using multiple sensor nodes 102, and based on the density of the network, the system can infer the trajectory of any moving object detected by the sensor node 102. As mentioned above, the thermopile sensor module 115 inside the sensor node is designed to measure temperature from a certain distance by detecting the infrared (IR) energy of the object. The higher the temperature, the more IR energy is emitted. The thermopile sensor module 115, which consists of small thermocouples on a silicon chip, absorbs energy and generates an output signal. The output signal is a small voltage that is proportional to the surface temperature of the object emitting IR in front of the sensor. Each thermopile sensor module 115 has 64 thermopiles, each of which is sensitive to the IR energy emitted by the object. To determine the trajectory, in various embodiments, each sensor module 115 divides the area captured by the sensor module 115 into a plurality of pixels organized in a rectangular grid in a direction aligned with the 64 thermopiles, each of the 64 thermopiles being associated with an 8×8 portion of the aforementioned grid. The system monitors the sequential changes in the temperature of consecutive pixels. The system determines that such sequential changes indicate the movement of the organism. The system records such movement as the formation of a trajectory in space. The more nodes in the network, the more accurate the inferences made about the trajectory, because the detected trajectory is not interrupted by "blind spots".

[0041] The calculation engine analyzes human behavior and trajectory. For example, with respect to occupancy control, the system can calculate the total number of people in a space to compare with established occupancy requirements. The system identifies all heat sources in the space monitored by the sensor module 115 and adds up the number of all heat sources generated by people.

[0042] With respect to occupant temperature screening, the system may detect the presence of a person by capturing the person's body heat. Once such detection is detected, temperature screening may include automatic adjustment of the sensitivity of the sensor module 115. It should be noted that body temperature screening may be different from body position detection. Body temperature screening means detecting an elevated body temperature of the detected person, resulting in higher sensitivity requirements than body position detection alone.

[0043] With respect to monitoring the body temperature of an occupant, the system 100 may be able to obtain the user's body temperature within a near field of one (1) meter from the sensor node 102 by using a more detailed 32×32 grid in the sensor module 115 to read the temperature of the area near the eye sockets. In order for the system to locate the eye sockets, the user may be required to gaze directly at the sensor node 102, thereby allowing the sensor module 115 to detect the pixel with the highest temperature.

[0044] With regard to analyzing the speed of movement of occupants, the system can record the movement of people under the network of sensor nodes 102. A series of "way points" are generated over time. The system uses the distance traveled based on the way point information and the time taken to travel the distance to calculate the movement speed of the user in question.

[0045] With respect to calculating total calories burned based on the occupant's movement, the user inputs information such as the occupant's weight, gender, and age to the system via interface 370. The system may use the captured movement speed and distance (as mentioned above) to calculate a rough calculation of calories burned during the time of the captured movement.

[0046] Behavioral analysis stems from the fact that by overlaying the structured network with the actual space, the captured data becomes contextualized. For example, the system can understand the shopping behavior of mobile bodies by cross-referencing the actual trajectories and dwell times captured by the sensor nodes 102 with the building floor plan that carries information about the location of specific products and aisles, such as Figure 5 and Figure 6 In particular, in various embodiments, user interface 370 allows a user to create a three-dimensional representation of the space in question, such as Figure 5 , Figure 6 and Fig. 9For example, the owner of a grocery store may record information about the location or products of an ice cream refrigerator by naming or "tagging" each sensor node 102 with the specific product located within the field of the sensor node 102. Thus, if sensor node #1 detects IR energy for 30 seconds, then the system determines that a person was within the field of sensor node #1 for 30 seconds. If sensor node #1 is tagged as being in front of the ice cream refrigerator, the system would provide data that a person was in front of the ice cream refrigerator for 30 seconds. An example of system output is shown in FIG. Figure 4A and Figure 4B shown.

[0047] In various embodiments, and as Figure 1A and Figure 1B As shown, each sensor node in the plurality of sensor nodes 102 can communicate with a module. The module can include, for example, an HTPA32D module or an HTPA16D module. The module can be a radio module. The module can be wireless. The module can be a hardware module.

[0048] The sensor node 102 may include a switch 120 (e.g., ALPS) that controls the power supply to the sensor node 102. The switch 120 may allow the manufacturer of the system to turn off the power supply to the sensor node 102 to save the battery 130 of the module during the entire transfer or transportation process from the manufacturer to the customer. After the sensor node 102 is delivered to the customer, the system can be installed with the switch 120 for the sensor node 102 turned on and left on. If the customer closes the store or system for a period of time, the customer can use the switch 120 to turn off the sensor node 102 to save battery life. The LED 125 on the sensor node 102 indicates the system status, such as, for example, an on mode and an off mode.

[0049] Figure 2A general data flow according to various embodiments is described in . The sensor node 102 can receive raw thermal data from the environment (step 205). The raw thermal data is compressed to create compressed thermal data (step 210). The gateway 135 receives the compressed thermal data from the sensor node 102. The gateway 135 decompresses the compressed thermal data to create decompressed thermal data (step 215). The cloud computing module 350 on the server receives the decompressed thermal data and creates detection data (step 220). The post-processing computing module on the server receives the detection data from the cloud computing module 350. The post-processing computing module processes the detection data to create post-processed detection data (step 225). The post-processing computing module sends the post-processed detection data to the database 360. The database 360 ​​uses the post-processed detection data to create time series detection result data (step 230). The system applies the background analysis algorithm to the time series detection result data to create analysis results (step 235). The system applies the API service 380 and the client APP to the analysis results to obtain 3D / 2D visualization of the data (step 240).

[0050] According to various embodiments, Figure 3A general system architecture including more details about the data flow is set forth in . In hardware, the sensor node 102 acquires raw data and then performs edge compression (step 305) and / or edge computing (step 310) to create Message Queuing Telemetry Transport (MQTT) raw data. The cloud computing module 350 on the server receives the MQTT raw data topic 1 from the sensor module 115 via the gateway 135. The cloud computing module 350 applies data concatenation and decompression (step 315) to the MQTT raw data topic 1 to create the MQTT raw data topic 2. The cloud computing module 350 applies the core algorithm (step 320) to the MQTT raw data topic 2 to create the MQTT result topic 1. The cloud computing module 350 applies world coordinate remapping (step 325) to the MQTT result topic 1 to create the MQTT result topic 2. The cloud computing module 350 sends the MQTT raw data topic 1, the MQTT raw data topic 2, the MQTT result topic 1, and the MQTT result topic 2 to the database 360. InfluxDB receives the data. The database 360 ​​applies background analysis to the data via a background analysis algorithm (step 330) to create background results (analysis results). The background results are stored in DynamoDB. DynamoDB also stores the sensor node 102 profile. The database 360 ​​can apply additional background analysis in response to updates or additional settings to the sensor module 115 profile. The API 380 obtains the background results from the database 360. The API applies real-time raw data, real-time detections, historical raw data, historical detections, historical occupancy, historical traffic, and / or historical duration to the data (step 335). The API sends the results to the user interface 370 (e.g., on a client device). The user interface 370 provides visualization (step 340) (e.g., Figure 4A and Figure 4B ). The user interface 370 also provides a settings interface (step 345). The settings interface may provide updates to the sensor module 115 profile. The user interface 370 also provides a login function (step 350). The login function may include AuthO / Firebase.

[0051] More specifically, the sensor module 115 may collect sensor module 115 data, pre-process the data, and / or send the collected sensor module 115 data to the gateway 135. The module may include an onboard microprocessor 140. The raw data from the sensor module 115 may be stored in the RAM of the microprocessor 140. The RAM serves as temporary storage for the system. The microprocessor 140 is configured to pre-process the raw data by eliminating outliers in the raw data.

[0052] Specifically, the microprocessor 140 applies a defined statistical procedure to the raw data to obtain processed data. In various embodiments, the module uses firmware software for preprocessing. The firmware software statistically determines the outliers of the temperature readings. For example, the outliers can be defined by normalizing the data by subtracting each pixel value from the average value of the frame. The result is divided by the standard deviation of the frame. Use a bilinear interpolation technique, that is, remove and replace pixel values ​​that are higher than three times the standard deviation or lower than one-third of the standard deviation with the product of the interpolation of adjacent values. Pixel values ​​are replaced instead of removed so that input detection is similar before and after the procedure. This technology helps to repair small minor data problems that may be caused by potential defects in the quality of the sensor module 115 data. The combination of firmware software, circuit design, and drivers enables the system to run an algorithm to determine the "region of interest" on each data frame in the data frame that represents human activity under the field of view of the sensor module 115. The region of interest is not a pixel with a specific temperature, but a pixel with a different temperature (higher or sometimes lower) relative to its surrounding pixels. The region of interest is then used to compress the processed data and prepare the compressed data for wireless transmission.

[0053] The system may include a rolling cache of ten frames of data for pre-processing. More specifically, the firmware of the microprocessor 140 may use the most recent ten frames of data captured to perform pre-processing and post-processing routines. Due to the limited amount of RAM memory on board (e.g., 8kb required for the application), the system may only process a subset of the data.

[0054] The data passed through the gateway 135 can be uploaded to the cloud computing module 350 on the server. The gateway 135 can be powered by any power source. In various embodiments, the gateway 135 is powered by a 110V outlet. The gateway 135 includes a module connected to a network (e.g., the Internet) via Ethernet, wifi, and / or cellular connections. In this way, the gateway 135 can upload data to any database 360, server, and / or cloud. The gateway 135 sends the pre-processed data and the compressed data to the computing engine in the cloud computing module 350, which in turn outputs the results to the database 360s. The gateway 135 pulls operating commands from the server to perform management functions, such as software updates, command modules to turn on and off sensor modules 115, change sampling frequencies, etc.

[0055] In various embodiments, the gateway 135 captures the compressed raw data in transit and sends the compressed raw data to an algorithm running on a processor (e.g., Raspberry Pi 4, model BCM2711), which then forwards the information to a server (e.g., cloud computing) for further processing. The processing of the data on the server includes decoding the compressed raw data, normalizing the temperature data of the sensor modules 115 according to the firmware and environmental settings of each sensor module 115, detecting objects, classifying objects, performing spatial transformations for world coordinate system positioning and fusion, multi-sensor module 115 data fusion, object tracking and trajectory generation, cleaning of outlier pixel-level readings, and other post-processing.

[0056] In various embodiments, the processing steps work using decompressed raw data. Decoding the compressed raw data optimizes data transmission and the consumption level of the battery 130. In addition, normalization of the temperature of the sensor node 102 to an appropriate temperature range enables the processing steps to adapt to various quality differences and environmental differences (these differences are expected from sensor nodes 102 located at different locations (spots) in space).

[0057] One of the core processing steps of the computing engine of the cloud computing module 350 is object detection and classification. This processing step detects the position of the object of interest in the frame and classifies the object into a person or an object of a different category (e.g., a laptop, a coffee cup, etc.). The spatial transformation from the local coordinate system to the world coordinate system makes the analysis "context aware". Using the spatial transformation, the system can compare and cross-reference the coverage of the spatial sensor module 115 with the actual floor plan and 3D model of the space in question. Multi-sensor module 115 data fusion unifies the data in the case of missing information or overlapping coverage between multiple sensor modules 115. As mentioned above, using various algorithms, object tracking and trajectory generation can distinguish multiple people from each other through time. Object tracking and trajectory generation provide a set of trajectories, derived from detected objects and personnel. The system uses such trajectories to determine behavioral analysis (e.g., the location and duration of stay), movement speed and direction. The post-processing step solves any minor inconsistencies in the detection and tracking algorithms. For example, when there are missing detections or gaps in the trajectory, the post-processing step helps to stitch the information together and repair any damaged trajectories.

[0058] The system may use a thermal application protocol interface (API), which may be located in an API layer in the system architecture, such as Figure 3The API hosts real-time and historical crowd statistics for spaces enhanced with the Sensor Module 115 solution. Built on REST, the API returns JSON responses and supports cross-origin resource sharing. The solution uses standard HTTP verbs to perform CRUD operations, and the API returns standard HTTP response codes for error indication purposes. In addition, namespaces are used to implement API versions, while token authentication is used to authenticate each API request. The API token on the dashboard is used to authenticate all API endpoints.

[0059] The token may be included in an Authorization HTTP header, prefixed with the string literal "token", with the two strings separated by a single space. If the correct Authorization header is not involved in the API call, a 403 error message is generated. HTTP Authorization header Authorization: Token YOUR_API_TOKEN. The endpoint uses standard HTTP error codes. The response includes any additional information about the error.

[0060] The API lists low-level "sensor module 115 events" for a sensor module 115 and a period of time. Included in each sensor module 115 event is a timestamp and a trajectory relative to the sensor module 115 in question. The trajectory does not have to equal any direction relative to space (e.g., entrance or exit). This call should only be used to test the performance of the sensor module 115.

[0061] This API provides information about the total number of entries into a specific space each day for the duration of a week. An analytics object is nested in the interval object for each result along with the data for that interval and the total number of entries. This call can be used to understand how many people visit a space on different days of the week.

[0062] The API records, counts, and lists all individual exits from a space of interest for a whole day (or any 24-hour period). Each result is accompanied by a timestamp and direction (e.g., -1). This call is used to find out when people leave a space.

[0063] The API provides information about the current and historical wait times at the entrance of a specific space at any given time during a day. An analytics object along with the data for that interval and the estimated total wait duration is nested in the interval object for each result. This call is used to find out how many people are waiting in line to enter a space at different time spans.

[0064] Webhook subscriptions allow callbacks to specified endpoints to be received on the server. For each space in which the event occurs, a webhook can be triggered after each event is received from one of the sensor modules 115. The system can create a webhook, get a webhook, update a webhook, or delete a webhook. When a webhook is received, the JSON data will be similar to the space and sensor module 115 events in the previous section. It will have additional information: the current count of the associated space and the ID of the space itself. The direction field will be 1, indicating an entrance, and -1, indicating an exit. If any additional headers are configured for the webhook, the additional headers will be included in the POST request. An example of received Webhook data would be a single event that occurred at a path connecting two spaces.

[0065] In various embodiments, the system may include one or more tools to help facilitate installation, help set up software and hardware, provide more accurate detection, create virtual representations, visualize human movement, test equipment, and / or troubleshoot equipment. The system may include any type of software and / or hardware, such as, for example, one or more applications, GUIs, dashboards, APIs, platforms, tools, network-based tools, and / or algorithms. The system may be in the form of downloadable software. For example, the software may be in the form of an application downloaded from a website that can be used on a desktop or laptop. The software may also include a web application that can be accessed via a browser. Such a web application may be device-independent and adaptive, so that the web application may be accessible on a desktop, laptop, or mobile device. The system may be obtained by licensing or subscription. One or more login credentials may be used to access the system partially or in full.

[0066] As used herein, a space may include the overall layout of an area that may be composed of one or more rooms. System functions may affect different spaces separately. The system may associate multiple rooms within a space. A room may include any walled part of a space (e.g., a conference room) or an open area within a space (e.g., a hot desk area or a corridor). Head count may include the number of people entering and exiting a space or room within a given time frame. Occupancy may include the number of people inside a room or space at a specific time. Fixtures may include furniture (e.g., chairs, tables, etc.) or equipment (e.g., washing machines, stoves, etc.).

[0067] Generally speaking, in various embodiments, the system can plan installation by, for example, visualizing sensor locations, visualizing coverage in a space using a 3D drag-and-drop interface, and / or understanding the number of sensors and / or hives that may be suitable for a room or space. The system can achieve more accurate detection by, for example, analyzing the spatial context to distinguish between humans and inanimate objects (and other confounding factors). The system can obtain the spatial context by receiving input about the layout of the space, such as 3D furniture, rooms, and markers / labels. Analysis of the spatial context may involve artificial intelligence and / or machine learning. AI is used to understand the space using the spatial context so that artificial intelligence can accurately identify human presence, behavior, posture, and other specific activities. The system can create a virtual representation of a real location (e.g., across any dashboard, setting an application that uses algorithms and APIs to visualize spatial data, and any other application). The virtual representation can receive tags, labels, and / or names for sensors, rooms, and spaces based on the tool. The virtual representation can be shown as a unique identifier in the dashboard and / or API.

[0068] In various embodiments, the system can visualize human movement by, for example, displaying the current frame and the previous frame (e.g., in the form of images such as dots), displaying and listing the location coordinates of sensors, virtual space layouts, and virtual fixtures within the background, displaying the trajectory of people, and / or displaying the posture of people (e.g., standing, sitting, lying down). The system can test and troubleshoot the device by, for example, showing the user what the sensors are detecting. The user can then confirm that the type and location of the actual object is related to the visual representation. The system can display what the sensors are detecting in a real-time and / or frame-by-frame representation of human presence and movement within the space. The system can also show when the sensors are online, offline, connected, and / or disconnected.

[0069] In various embodiments, the system may include the ability to create a spatial layout. Creating a spatial layout may involve adding a space and adding a space name. The system will then store the space with its name. The system provides users with the ability to create and manage multiple spaces. It may be useful to have several separate spaces when monitoring multiple floors in a building (e.g., first floor, second floor), facilities with multiple separate rooms (e.g., senior citizen apartments), or multiple facilities located in different physical locations (e.g., Boston laboratory, San Francisco laboratory). In various embodiments, as part of the settings, the system may provide the ability to rename the space, add automatic alignment, add visual smoothing, add display local detection, add toolbars (e.g., main toolbar, side toolbar, etc.), add fixtures from the library (e.g., pieces of furniture from the furniture library), or return to the project library. The main toolbar may include functions related to rooms, sensors, boxes, languages, and / or saves. Exemplary side toolbar functions may include 2D or 3D, display or hide sensors, display or hide rooms, display or hide fixtures, etc. Toolbars and features may be described as being located on the main toolbar or on the side toolbar, but any feature can be associated with any toolbar.

[0070] In various embodiments, the system may include functionality for adding rooms to match or resemble a floor layout. The system may display a control panel (e.g., in response to selecting a room). The control panel may allow for changing the size, marking certain locations or features, and / or selecting a border color for each room. In response to selecting a room icon, the system may add one or more rooms to the space. In response to automatic alignment being activated, objects being moved (e.g., fixtures, sensors, or rooms) may automatically snap to the edges of nearby similar objects. For example, a chair may be automatically moved next to another object to make it easier and faster for a user to arrange objects in a neat and orderly manner to match a floor plan. The system may determine that the object being moved is of the same type as an existing object based on similar identifiers or tags associated with each of the objects. In response to automatic alignment being deactivated, a user may freely and incrementally move objects that may not be aligned with similar objects.

[0071] In various embodiments, the system may include functionality for adding fixtures to a virtual space in the GUI (which may be related to fixtures present in the physical space). Fixtures may include, for example, furniture or equipment that a user may add to a space. Fixtures may allow a user to distinguish rooms and allow a user to contextualize the movement seen on the screen. The system may allow a user to add fixtures by selecting any of the furniture or equipment icons and then dragging and dropping the fixtures to specific locations in different rooms. The system also provides a user with functionality for virtually selecting a fixture and then deleting or rotating the fixture using panel controls. The system also provides a user with functionality for virtually adjusting the size or position of a fixture. In various embodiments, a user may enter specific coordinates of a piece of furniture so that the system understands the size and position of the furniture. In addition, when a user moves a fixture, the system may display the distance between the center point of the fixture and each of the four walls of the room in which the fixture is placed. The coordinates of the furniture relative to the entire space may be stored in the system via an API. In this way, a user may extract this information from the backend as needed. A user may add any number of fixtures to a space or room as needed. A user may stack fixtures or furniture on top of other fixtures or furniture. Additionally, if the physical table is particularly large, the user can use multiple virtual tables to match its size. The system encodes, identifies, stores, and takes into account the presence and coordinates of fixtures. The system uses this data when determining whether a detection is human and whether the detection should be counted in the occupancy or headcount of a room or space. The system includes functionality (e.g., using an API and an algorithm) to encode each fixture (e.g., table, door, etc.) using a fixture type. The user can select a fixture and a fixture type from an icon. The API can save the fixture and its coordinates on the system so that the algorithm can use this information to identify detections and behaviors. In response to a location selected by the user, the system records the xy coordinates of the center point of the fixture, the type of fixture, and the rotation from the center point in degrees (e.g., 0, 90, 180, 270 degrees). Rotation can include the action of rotating an object (room, sensor, fixture) by 0, 90, 180, 270 degrees from its center point. Rotation can be achieved by selecting a circular arrow to rotate the object. Fixtures in the system can be set to be in a default pointing direction that may be different from the orientation of the real furniture in the room. The user can rotate the virtual furniture model to 4 different orientations (i.e., 0, 90, 180, 270 degrees from its default orientation) along the center point of the model.

[0072] The system can show when people are on, around or passing through pieces of furniture. The system stores names or icons associated with different fixtures. These names or icons include factors or rules that the system considers when analyzing fixtures. For example, a bed icon can include a rule that humans can be on the bed, while a table icon can include a rule that humans will not be on the table. Examples of other fixtures (with rules that humans will not be on fixtures) include tables, counters, stoves, refrigerators, dishwashers, sinks, radiators, counters and / or washing machines. Examples of other fixtures (with rules that humans can be on or in fixtures) include beds, sofas, chairs, toilets and / or showers. The system can infer the activities of people based on the amount of time they are next to furniture. For example, if the system detects that a person is near a TV for an hour, the system can infer that the person likes the shows played on TV. The location and inference information also provides valuable context for the algorithm to allow the system to infer daily activities and enable more accurate human detection. For example, for daily activities, if a detected object (e.g., represented by a purple sphere) is located on or within the outline of a bed fixture, the system can infer that a human is sleeping. The system can then infer the person's sleep time by determining how long the person is on the bed fixture. As another example, for more accurate human detection, the system can detect both human heat sources and non-human heat sources (such as stoves and laptops). If the detected heat source is found to be located in the middle of a fixture such as a table, the system can identify that this is not a human. Therefore, the system may not count this detection as part of the occupancy data (e.g., the user may receive the occupancy data via an API or dashboard). The system may include functionality for activating or deactivating a "show local detection" option to view or hide the detection (e.g., purple sphere).

[0073] In various embodiments, the system may include a "visual smoothing" feature. Disabling "visual smoothing" causes the detection sphere to be displayed at specific coordinates, frame by frame, exactly as it is detected. Activating "visual smoothing" causes the frame-by-frame movement of the purple sphere to be displayed as a smooth, continuous animation. More specifically, when a sensor detects an object in physical space, the system may create an "agent" (e.g., a purple sphere) that appears at the corresponding location in virtual space. The system may also assign a "lifetime" to the agent. The "lifetime" may be the length of time that the purple sphere appears in virtual space to show the detection. In various embodiments, this lifecycle may be set to last 300 milliseconds, which matches (or is similar to) the interval at which the sensor sends a new detection. The system uses new detections of new locations in the physical world to update the position of the purple sphere in virtual space. As mentioned, the system may display the purple sphere frame by frame or with visual smoothing. The purple sphere may appear to be flickering, but the sphere may actually be displaying real-time detections captured by the sensor every 300 milliseconds. The flickering effect is caused by the fact that the purple sphere changes from opaque to transparent during its 300 millisecond lifecycle. If the sphere is more opaque, the detection is likely newer. If a person is standing still under the actual sensor, the purple sphere will appear to flicker in place. If a person is moving under the actual sensor, and visualization smoothing is not activated, the system may display a string of purple spheres. For example, each of the purple spheres may continuously go from opaque to transparent at each detection coordinate (or pixel) every 300 milliseconds. With visualization smoothing activated, only one purple sphere appears on the screen, and it appears to move linearly. During the 300 millisecond lifecycle, the system searches for the next detection within a 1-foot radius and "refills" its lifecycle to another 300 milliseconds. This means that the spheres continue to appear, moving from one point to another according to the detection coordinates (pixels).

[0074] In various embodiments, the system may include functionality for adding one or more boxes and / or thermal sensors. Data from outside the room may still be stored via the API, but data from outside the room may not be displayed on the dashboard. The dashboard displays room-specific activity and occupancy within the space. Therefore, the system instructs the user to place sensors and objects within the room. The box may provide a gateway function to connect to the thermal sensors and transfer data from the thermal sensors to a storage location (e.g., the cloud). For example, a box may be connected to a certain number of sensors (e.g., up to 12 sensors), and then additional boxes may be required for additional sensors. For ease of installation, each box may be pre-configured with a set of sensors (e.g., the sensor ID is loaded as sensor data into the box's database). The pre-configuration process may include two parts. The sensor can be set to have the same NetID as the box. This enables the sensors to connect to each other. The box can obtain the sensor MAC address and sensor mode programmed into the configuration file. This enables the box to properly manage the sensor frame rate. The system may allow the user to add one or more boxes to the space. The system may include functionality to add a box to a space by scanning a code (e.g., a QR code) or by entering a box ID (e.g., an input box ID found on a sticker under the box). During the process of adding the box data to the system, the system also receives preconfigured sensor data (because the box data is preconfigured to include sensor data). The system uses the sensor data to display a "sensor" icon for each box. The system can display sensors from different boxes because the sensors can be color-coded to show that the sensor belongs to a specific group and box. The specific implementation may vary, but it can generally be visually identified by the system. In response to receiving a selection of a sensor icon for a box, all sensors associated with the box will be displayed so that each sensor can be added to the space. The user can drag and drop each virtual sensor somewhere in the room to record room occupancy, or drag and drop it on a doorway to record the number of people entering and exiting the doorway. Each sensor can be unique and can be identified by a unique address (e.g., a MAC address). As such, the user should place the virtual sensor at the same location as the location of the corresponding physical sensor, at the same orientation as the orientation of the corresponding physical sensor, and in the same room as the room of the corresponding physical sensor.

[0075] In various embodiments, the system may include functionality for setting up and / or calibrating virtual thermal sensors. Each virtual thermal sensor may appear on the display (e.g., as a square). In response to the virtual sensor being turned off, the virtual sensor may appear in some manner (e.g., a black square) to indicate that the actual sensor is not detecting anything. In response to the virtual sensor being turned on, the virtual sensor is displayed as a grid (e.g., an 8×8 grid consisting of 64 squares). Each of the 64 squares may represent each pixel, and the color of each pixel may represent the temperature detected by the actual sensor at that point in the grid. The color of each pixel may include a range of chromaticities to indicate the temperature level. For example, the range of chromaticities may be from yellow (lower temperature) to red (higher temperature).

[0076] In response to receiving the selection of the virtual sensor, the system displays a control panel (e.g., located on the left side of the display). Using the control panel, the system can allow the user to set the virtual sensor height. For example, it may be important that the aisles in the grocery store are within the field of view, but the janitor's closet may be located outside the field of view. The optimal (and preferred maximum) height of the actual sensor is about 3.2 meters. Such a height can provide the maximum coverage and optimal resolution required for human detection. The optimal resolution can be based on the ability of the algorithm to detect human presence from the thermal image from the sensor. The optimal resolution refers to the resolution at which the system can accurately and reliably detect human presence. The height of the virtual sensor corresponds to the height of the ceiling or the height on the wall to which the actual sensor will be attached. The higher the sensor is above the floor, the wider the coverage of the floor by the sensor. The lower the sensor is from the floor, the narrower the coverage of the floor by the sensor. Based on the height, the system can determine how big the floor space that the sensor is monitoring (covering) is.

[0077] The system can use the formula (90%*2*tan(30°)*height)^2. For example, a ceiling height of 3 meters can result in a coverage area of ​​3.03m×3.03m on the floor. The standard supermarket ceiling height is 5.78m, the standard office ceiling height is 3.12m, and the standard door height is 2.43m. The system also allows the user to test different heights to confirm that certain areas are within the sensor's field of view (or outside the field of view). A sensor height of 110 inches (2.8m) can provide an effective coverage width of 106 inches (2.7m), a sensor height of 102 inches (2.6m) can provide an effective coverage width of 78 inches (2.0m), a sensor height of 95 inches (2.4m) can provide an effective coverage width of 63 inches (1.6m), and a sensor height of 87 inches (2.2m) can provide an effective coverage width of 56 inches (1.4m).

[0078] In various embodiments, the system may also allow the user to set the virtual sensor orientation to match the way the actual sensor is physically mounted to ensure accurate representation between the virtual world (e.g., in the settings application) and the physical world. The sensor orientation in both the real world and the virtual world must be similar so that visible detections also match. For example, a person standing in the northeast corner of a room must appear in the northeast corner of the corresponding virtual sensor pixel in the settings application. To help match sensor orientation, the physical sensor can include an arrow (e.g., on its mounting plate). When the user adds the physical sensor to the settings application as a virtual sensor, the user matches the direction of the virtual sensor arrow to the direction of the physical sensor arrow by rotating the virtual sensor.

[0079] In various embodiments, the system may include a function to view detections. "Detection" refers primarily to the detection of human presence. The actual sensor may capture a thermal map of a region at, for example, 3 to 5 frames per second. The system may detect human presence by identifying areas of the thermal map that are related to human body temperature (or "thermal signature"). The system may indicate detection of a person on a display (e.g., a purple sphere). The average normal body temperature of a person is generally accepted to be 98.6°F (37°C). However, normal body temperature may have a wide range, from 97°F (36.1°C) to 99°F (37.2°C). Temperatures above 100.4°F (38°C) may mean that the person may have a fever due to an infection or disease.

[0080] In various embodiments, the detection process may include sensitivity adjustment. The system may receive data about the temperature in a space or room based on a thermometer located in the physical space or room. Sensitivity adjustment may involve improving the system's ability to detect human presence in an environment where its temperature is close to that of the human body. Detection is improved by changing parameters involving the temperature difference between the detection (human temperature) and the surrounding environment. Improving sensitivity may involve minimizing this temperature difference so that the system can more easily detect the presence of humans in, for example, very warm climates. In other words, in a colder climate of 65 degrees, the system may more easily determine that any object that is 30 degrees or more above the general room temperature is a human. However, in a warmer climate of 96 degrees, the system may need to detect an object with a temperature 2 to 3 degrees higher than the room temperature before the object can be considered a human.

[0081] This detection information can be further processed by the system algorithm to double-check and ensure that the data ultimately sent to the API and dashboard is accurate. This further processing can include additional criteria to filter out detections that do not behave like humans. The system can filter out any objects with a temperature below the human body temperature range (e.g., from 97°F (36.1°C) to 99°F (37.2°C)). Filtering can include detections that do not move at all (e.g., appliances like stoves) or stationary detections that appear to be located on fixtures where humans are not expected to be located (e.g., in the middle of a table). For example, the system may determine that a heat map displayed on a table is more likely to indicate a laptop. In addition, the system can store coordinates around each fixture so that detections of objects with coordinates that overlap with the coordinates of any fixture are not counted. In other words, the coordinates around the fixture are blacklisted so that any detections within these coordinates are not counted.

[0082] In various embodiments, detection may include head counting. A sensor that determines head count based on people entering and leaving a room (a head count sensor) may be mounted above an entry doorway to the room (e.g., on a wall facing the interior of the room). The head count sensor may use data associated with a virtual threshold (or door casing), which may be a certain distance from the door. The head count sensor may only count detections of people crossing the door casing. For example, an "In" is when a person crosses the door casing from left to right, while an "Out" is when a person crosses the door casing from right to left. The door casing reduces false readings of people who, for example, may stick their head inside a door to see what is inside the room, but who never fully enter the room.

[0083] In various embodiments, the system may include functionality for 2D or 3D view settings. In response to receiving a selection of a 2D / 3D button, the system may display all (or any portion) of the entire space in 2D or 3D. A 3D view may provide a more "real" spatial context for users who are not familiar with floor plans. Such a view may also provide a more intuitive way to understand spaces, detections, and data. In various embodiments, the system may include functionality for showing or hiding various features (such as, for example, sensors, rooms, or fixtures). The system may include functionality for editing spaces, managing spaces, different viewing modes for visually communicating data, and the like.

[0084] In various embodiments, the system can visually convey data about the flow of people and / or dwell time over a certain period of time. The flow of people can be conveyed based on the number of people who "enter" and "exit" a given doorway (or across the door frame) and enter a room or space. The flow of people can also be conveyed by the path or trajectory line that people move in the space. The system can include a head count view, where the number of in and out can be displayed on a virtual layout. The system can also include a trajectory view that displays multiple detections over a period of time, thereby forming a detection stream in the display. The detection stream can be used to determine the path of the flow of people. The system can also display a linear movement trajectory, where the system creates lines through multiple detections to create lines representing the path of the flow of people. The dwell time can be the amount of time a person spends in a set of coordinates in a room or space. The dwell time can be determined by measuring the amount of time a person is detected in a certain area. The system can infer or identify a unique detection (the same person) by location and trajectory, even if the system cannot determine whether it is the "same" person previously detected. Dwell time can be displayed as a heat map, where darker colors can indicate more time spent in a particular area, and lighter colors can indicate less time spent in an area.

[0085] As discussed above, traditional computer vision uses high-resolution images to develop human gesture detection algorithms. High-resolution images can include, for example, recorded high-resolution video clips. Many of the existing algorithms detect key points (key points can be points on the human skeleton) to detect human gestures from images, and the detection of key points requires a high-resolution camera system. As such, the detection of key points using a reduced-resolution system may not be feasible. However, a system that can use reduced-resolution images may be important for promoting more privacy.

[0086] In various embodiments, the system can provide functionality for gesture detection by using a gesture detection algorithm on a reduced resolution image. For example, the system can display whether a person is standing, sitting, lying down, or has fallen. In general, the algorithm can include a submodule for extracting a bounding box containing human pixels. The algorithm can also include a submodule for detecting the gesture of a human inside the bounding box. These algorithms can be algorithms based on deep neural network learning, which can be data-driven. The neural network can be a CNN (convolutional neural network), which is an artificial neural network specifically designed to process pixel data for image recognition and processing.

[0087] Generally speaking, defining the boundaries of a bounding box is one of the behaviors learned by a neural network. The neural network can be trained on an image with a bounding box that can contain all pixels (or a subset of pixels) corresponding to a human in the image. The neural network can use this information to predict where (multiple) bounding boxes should be on the image. More specifically, in various embodiments, the system can receive an image of a human. The input image / frame can be low resolution, such as, for example, an 8×8, 32×32, or 64×64 pixel image. For each frame, a human annotator can first observe the posture of a human (e.g., a test subject) in the image, mark the posture and draw a bounding box around the person in the image. The human annotator can confirm that a human is present in the image by viewing the recorded high-resolution video / image. However, such high-resolution images may not be used to train the algorithm.

[0088] In various embodiments, a user may label a posture with an integer. A user may enter an integer for a particular posture on the screen. The GUI may include a text box with a field that accepts an integer for one or more postures. The system may associate the integer with a posture in the database. For example, label 0 may represent a sitting posture, label 1 may represent a standing posture, label 2 may represent a lying posture, and the like. In various embodiments, a user may draw a bounding box around a human in an image on the screen (e.g., using any type of device that accepts input on the GUI) so that the bounding box is stored as an (x, y) coordinate that the system can recognize. The system may use such labels and manually annotated data (e.g., bounding boxes) to train a learning algorithm. The learning algorithm may be trained using, for example, training based on gradient descent. Based on the training, the learning algorithm may learn how to automatically create bounding boxes, collect data from within the bounding boxes, and determine human postures from the collected data. The collected data used by the algorithm may be collected over time or during an initial calibration session while the system is running. The system may obtain environmental data from a specific environment in which sensors are deployed, so the system may use environmental data to adjust its algorithm based on the specific environment. Data from the environment may include parameters and / or variables from the environment, such as, for example, ambient temperature, room temperature, floor plan, non-human thermal objects, gender of the human, age of the human, height of the installed sensor, clothing of the human, body weight of the human, etc. Pixel data, thermal data, and / or environmental data collected from the environment may be used to train the algorithm through machine learning or artificial intelligence.

[0089] Due to the low resolution, it may be difficult to determine the difference in image intensity between different pixels, so the algorithm may not attempt to determine features based on image intensity. Instead, the algorithm may focus on distinguishing features from the characteristic patterns of overhead thermal signatures of human presence. CNNs may implicitly define what constitutes a "distinguishing feature." Each layer of the network defines a "feature map" learned through stochastic gradient descent. The user may not be aware of the features that the network considers important. The system can discover edges, curves, sharp contrasts in adjacent pixel values, and so on. However, due to the nature of neural networks, the system may not have any (or little) information about which features are important for distinguishing postures.

[0090] As discussed above, in various embodiments, the system can extract a region that includes distinguishing features of a human (e.g., the region is represented as a rectangle representing a bounding box). The bounding box can limit the amount of data analyzed by the system, and the box forces the algorithm to focus only on human outlines. In this step of the algorithm, the system can focus on the extracted bounding box in an attempt to classify the human pose. The system can extract human pose information for each frame. A frame can be a single image captured by one of the thermal cameras.

[0091] In various embodiments, the system may calculate "aggregated postures" to help smooth posture changes across multiple frames over a period of time. For example, aggregated postures may be determined based on a pattern of a collection of postures collected over a given period of time (e.g., the most frequent postures in the collection). All postures in the collection may be different, but the posture aggregation method creates a consensus. This consensus may be referred to as an "aggregated posture." For example, the system may obtain a posture consensus every 5 seconds. Aggregated postures may include, for example, sitting down, standing up, or lying down. The system may determine that an event is a fall based on certain changes in posture or that the posture lasts for a certain amount of time. For example, if an event consists of the following: (i) the aggregated posture changes from standing / sitting down to lying down, and (ii) the lying posture lasts for a certain amount of time (e.g., at least 30 seconds), the event is determined to be a fall.

[0092] Using pattern recognition, fuzzy logic, artificial intelligence, and / or machine learning, the system can determine human postures based on similar thermal signatures. In various embodiments, with respect to pattern recognition, the system can extract distinguishing features and / or patterns from frames to help identify the postures contained in the frames. In various embodiments, with respect to artificial intelligence, and based on the collected human annotation data and / or frames, the system can automatically learn these patterns by using CNN. As illustrated in FIG. 11 , the patterns from human thermal signatures can be different, and different patterns can cause the system to classify human postures differently. For example, Fig.11AA thermal signature indicative of a standing position is shown, Fig. 11B shows a thermal signature indicative of a sitting position, and Fig. 11C A thermal signature indicative of a lying position is shown.

[0093] like Fig.12 As described, in various embodiments, the system can perform a pose inference process. The system can obtain an input image (e.g., an 8×8, 32×32, or 64×64 pixel image) from an infrared sensor (step 1205). The upper and / or lower limits of the resolution can be based on privacy issues, sensor power consumption, data cost, data bandwidth, computational cost, and / or computational bandwidth. The system can apply a CNN to the input image (step 1210). The CNN can learn different filters at each subsequent layer of the network through stochastic gradient descent, thereby implicitly extracting important information from the image.

[0094] The system can create a vector of image features (step 1215). The vector can be an intermediate result composed of learned image features output by a CNN. A transformer encoder / decoder (step 1220) can be used to convert the vector of image features into box predictions (step 1225). DETR (DEtectionTRansformer) can use a traditional CNN backbone to learn a 2D representation of the input image. Before passing the image to the Transformer encoder, the model may flatten the image and supplement the image with a positional encoding. The Transformer decoder can then take a small fixed number of learned positional embeddings (e.g., object queries) as input and additionally join the encoder output. The system can pass each output embedding of the decoder to a shared feedforward network (FFN), which can predict a detection (e.g., class and bounding box) or a "no object" class.

[0095] In various embodiments, the system may include a box prediction that may predict the size and position of a bounding box that includes distinguishing features of a human. The system may obtain the interior of the bounding box (step 1230). The system obtains the interior of the bounding box because anything that falls outside the bounding box may not be of interest to the system because the system is interested in recognizing human postures of humans. In various embodiments, the system applies a CNN to the interior of the bounding box (step 1235). More specifically, the CNN applied to the interior of the bounding box may implicitly learn features of the image given as training. These features may be invisible to human users and may never be explicitly expressed by the neural network. Based on the CNN, the system creates a posture prediction for the image (step 1240). More specifically, during the training phase of the model, the system may extract features to distinguish between human postures. This is the prior knowledge used during the inference / testing phase when using the training model to distinguish between various postures. Posture predictions may include, for example, sitting, standing, lying, or any other posture or configuration of a human. The system can also detect other postures using higher resolutions (32×32 pixels or higher). For example, the system can use higher resolution to achieve better performance, allowing the system to distinguish between standing and sitting postures. The system can also distinguish between activities such as exercising, dancing, running, eating, etc.

[0096] like Fig.13 As described, in various embodiments, the system may perform a fall detection process. The system may acquire images (step 1305). For example, all images may be aggregated within 5 seconds. The system may benchmark the amount of time used to aggregate the images. If the amount of time used to aggregate the images is too short, the system may lose image details. If the amount of time used to aggregate the images is too long, the system may acquire noisier data in the images. The system may not make any assumptions about the location of the person in the frame. The person may be stationary or moving. The system acquires data about the posture of the person in each frame. The above posture inference process is performed on the images, such as Fig.12 As described (step 1310). The result of executing the pose inference algorithm for each frame is the pose prediction for that particular frame. After pose inference, the system can obtain many predicted "poses". Fig.12 It is shown as "postures collected within 5 seconds". After passing through the "posture aggregation" module, the system can determine an "aggregated posture", whether it is standing / sitting or lying down. The system can then use this information to further determine whether a fall has occurred.

[0097] In various embodiments, the system may perform posture aggregation (step 1315). More specifically, the system may determine the aggregated posture based on the pattern of the collection of postures collected within a given time period. For example, the posture that appears most frequently in the collection. Although all postures in the collection may not be the same, the posture aggregation method creates a consensus, that is, the aggregated posture. The system may determine the aggregated posture (step 1320). The aggregated posture may be determined to be standing or sitting (step 1325). The system may determine that the aggregated posture is lying (step 1330). If the aggregated posture is lying, the system will check the database of previously aggregated posture data and determine whether the aggregated posture at the previous timestamp is standing or sitting. The system can compare the posture with the posture recognized in the previous frame. The frames can be sorted by time. The system can determine that the aggregated posture changes from standing or sitting to lying, and then the posture remains in the lying position for at least 30 seconds (step 1335).

[0098] The system can store timestamps associated with each of the various actions. The system can check the difference between the start timestamp and the current timestamp to determine if the time difference exceeds a threshold. The threshold can be pre-specified, predetermined, dynamically adjusted, algorithm-based, etc. In various embodiments, if the fall exceeds a threshold amount of time, the system issues a fall alarm. The fall alarm can be sent to any other device via any communication method. For example, the system can send a signal to an application on a relative's smartphone via the Internet to notify the relative that a fall may have occurred. In addition to the alarm, the system can also provide data about people, locations, facilities, health information, demographic profiles, fall history, etc.

[0099] Due to the nature of CNNs, CNNs can assign a certain type of pose. However, the system can ignore some frames if it is unsure about the pose or the frame. For example, the system might use a confidence score of 0.7. A confidence score of 0.7 can indicate that if the system predicts with 70% confidence, then the pose is a fall to a certain extent. The system can also predict a fall with more than 0.7 or less than 0.7 confidence.

[0100] The system can determine a potential fall (step 1340). For example, if a person lies on a bed or a sofa, the system can determine such action as a "potential fall." However, lying on a bed or a sofa can be a normal action, and such action may not be a true fall. As such, the system can perform post-processing on the potential fall (step 1345). In various embodiments, the post-processing can include filters, such as, for example, blocking some spatial locations where a fall could not reasonably occur (e.g., a bed). Another filter can include a confidence threshold for the detection of a potential fall using an algorithm.

[0101] Based on post-processing, the system can confirm the potential fall as a confirmed fall (step 1350). In particular, after using one or more filters, the system can confirm the fall and provide a notification of the confirmed fall. For example, in the user interface / software product, the user can create and place virtual furniture (e.g., bed, chair, closet, etc.) or occluded areas in the settings application. Even if a "potential fall" is detected according to the machine learning algorithm, the system can automatically blacklist these areas to prevent the triggering of a fall alarm. For example, as discussed above, if a person lies down or sleeps in bed, the machine learning algorithm can detect a "possible fall". Because lying down or sleeping in bed is a normal action, such an action will not trigger a fall alarm for the user. However, if the person actually falls on the floor, the machine learning algorithm can detect a "potential fall". The system then determines whether the potential fall is within the blacklist area or outside the blacklist area. If the potential fall is located in the blacklist area, the system will not issue an alarm. If the potential fall is outside the blacklist area, the system can determine that the fall is a "confirmed fall" and trigger (e.g., send) a fall alarm for the user.

[0102] The posture detection system can include many commercial applications and business benefits. For example, for senior living communities, the system can provide predictive insights and prescriptions. In this regard, the system can provide flags or notifications to caregivers so that intervention can be made as early as possible. Specifically, the system can measure and track frailty based on, for example, analysis of baseline movement patterns and changes in frailty. The system can also detect and / or flag abnormal activities (e.g., staying in bed or bathroom for too long). The system can also detect and / or flag trips to the bathroom at night.

[0103] Previous systems typically required active steps and the use of wearable devices or the completion of surveys. However, current systems can offer a value proposition of privacy and non-intrusion. The system can also passively sense actual behavior without considering changes in behavior.

[0104] Currently, frailty is measured in a clinic or doctor's office. The doctor can perform a test on the patient that includes a series of activities that may take 5 to 10 minutes to complete. However, such a quick test does not provide a comprehensive understanding of the patient, and the test is performed in an artificial context, where the patient may be more focused, more diligent, etc. In this regard, frailty scores may vary over time due to different scenarios and different efforts made by the patient. The current system improves on the prior art frailty testing by using "longitudinal" tracking, which includes tracking the movement of a person over time. In this way, the current system is able to provide a more comprehensive understanding of frailty over time.

[0105] Using the gesture detection function, the system can provide notifications or reports about events (e.g., falls, abnormal behavior, etc.) to, for example, nursing staff, building management systems, alarm systems, notification systems, and / or emergency response systems. The report can include data associated with the event (e.g., fall), such as, for example, location, time of day, actions before and / or after the fall, nearby items (e.g., furniture), items held by people (e.g., groceries, walkers, canes, other people), etc. Compared with radar or other sensor devices, the system can be cheaper, faster to install, and easier to install. The system can also analyze images to facilitate audits and / or compliance. For example, the system can use images over a period of time to monitor or audit the care provided (e.g., bed checks were completed at 11 pm the previous night). The system can also use images over a period of time to measure the time spent providing care (e.g., the average number of minutes in the bathroom with the patient every day). The system can also be integrated with other systems such as scheduling systems and reporting systems. For example, the system can obtain data about when an employee starts a shift, when an employee ends a shift, employee names and / or identifiers, and the time claimed by an employee to help a patient. The system may compare such submitted data with actual data obtained from images of resident activity and / or resident care to determine the accuracy of such submitted data.

[0106] The detailed description of various embodiments herein refers to the drawings and pictures, which illustrate various embodiments by way of illustration. Although these various embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure, it should be understood that other embodiments can be implemented and logical and mechanical changes can be made without departing from the spirit and scope of the present disclosure. Therefore, the detailed description herein is presented for the purpose of illustration only and not for the purpose of limitation. For example, the steps listed in any method or process description can be performed in any order and are not limited to the order presented. In addition, any function or step can be outsourced to one or more third parties or can be performed by one or more third parties. Without departing from the scope of the present disclosure, the systems, devices and methods described herein can be modified, added or omitted. For example, the components of the systems and devices can be integrated or separated. In addition, the operation of the systems and devices disclosed herein can be performed by more, less or other components, and the described methods can include more, less or other steps. In addition, these steps can be performed in any suitable order. As used in this document, "each" refers to each member of a set or each member of a subset of a set. In addition, any reference to the singular includes multiple embodiments, and any reference to more than one component can include a singular embodiment. Although specific advantages have been enumerated herein, various implementations may include some, none, or all of the enumerated advantages.Systems and methods are provided.

[0107] In the detailed description herein, references to "various embodiments," "one embodiment," "embodiment," "example embodiment," etc. indicate that the described embodiments may include specific features, structures, or characteristics, but each embodiment does not necessarily include the specific features, structures, or characteristics. In addition, these phrases do not necessarily refer to the same embodiment. In addition, when a specific feature, structure, or characteristic is described in conjunction with an embodiment, whether or not explicitly described, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in conjunction with other embodiments. After reading the description, it will be clear to those skilled in the relevant art how to implement the present disclosure in alternative embodiments.

[0108] Benefits, other advantages and solutions to problems have been described herein with respect to specific embodiments. However, benefits, advantages, solutions to problems, and any elements that may cause any benefits, advantages or solutions to appear or become more apparent should not be interpreted as key, essential or basic features or elements of the present invention. Therefore, the scope of the present invention is limited only by the appended claims, wherein unless explicitly stated otherwise, reference to an element in the singular does not mean "one and only one", but "one or more". In addition, when a phrase similar to "at least one of A, B or C" is used in a claim, it is intended that the phrase be interpreted as meaning that A can exist alone in an embodiment, B can exist alone in an embodiment, C can exist alone in an embodiment, or any combination of elements A, B and C can exist in a single embodiment; for example, A and B, A and C, B and C, or A and B and C. In addition, regardless of whether an element, component or method step is explicitly recorded in a claim, any element, component or method step in the present disclosure is not intended to be dedicated to the public. Any claim element herein shall not be interpreted according to the provisions of 35 U.S.C. 112 (f) unless the element is explicitly recorded using the phrase "device for..." As used herein, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0109] A computer program (also referred to as computer control logic) is stored in a main memory and / or a secondary memory. The computer program may also be received via a communication interface. Such a computer program, when executed, enables a computer system to perform features as discussed herein. In particular, the computer program, when executed, enables a processor to perform features of various embodiments. Therefore, such a computer program represents a controller of a computer system.

[0110] These computer program instructions may be loaded onto a general purpose computer, a special purpose computer, or other programmable data processing device to produce a machine, such that the instructions executed on the computer or other programmable data processing device create a device for implementing the functions specified in one or more flowchart blocks. These computer program instructions may also be stored in a computer readable memory, which may direct the computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the functions specified in one or more flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing device to cause a series of operating steps to be executed on the computer or other programmable device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flowchart blocks.

[0111] In various embodiments, the software can be stored in a computer program product and loaded into a computer system using a removable storage drive, a hard disk drive, or a communication interface. When executed by a processor, the control logic (software) causes the processor to perform the functions of the various embodiments as described herein. In various embodiments, the hardware components can take the form of an application specific integrated circuit (ASIC). The implementation of hardware to perform the functions described herein will be apparent to those skilled in the relevant art.

[0112] As will be appreciated by one of ordinary skill in the art, the system may be embodied as a customization of an existing system, an add-on product, a processing device that executes upgraded software, a stand-alone system, a distributed system, a method, a data processing system, an apparatus for data processing, and / or a computer program product. Thus, any portion of the system or module may take the form of a processing device that executes code, an Internet-based implementation, a fully hardware implementation, or an implementation that combines Internet, software, and hardware aspects. Additionally, the system may take the form of a computer program product on a computer-readable storage medium having a computer-readable program code device embodied in the storage medium. Any suitable computer-readable storage medium may be utilized, including hard disks, CD-ROMs, BLU-RAY Optical storage devices, magnetic storage devices, etc.

[0113] In various embodiments, the components, modules and / or engines of the system may be implemented as micro-applications or micro-apps. Micro-applications are typically deployed in the context of a mobile operating system, including, for example, Mobile operating systems, operating system, operating system, The micro-application can be configured to utilize the resources of the larger operating system and related hardware via a set of predetermined rules for managing the operation of various operating systems and hardware resources. For example, in the case where the micro-application expects to communicate with a device or network other than a mobile device or a mobile operating system, the micro-application can utilize the communication protocol of the operating system and related device hardware under the predetermined rules of the mobile operating system. In addition, in the case where the micro-application expects input from the user, the micro-application can be configured to request a response from the operating system, which monitors various hardware components, and then transmits the detected input from the hardware to the micro-application.

[0114] The systems and methods may be described herein in terms of functional block components, screen shots, optional selections, and various processing steps. It should be understood that such functional blocks may be implemented by any number of hardware and / or software components configured to perform the specified functions. For example, the system may employ various integrated circuit components, such as memory elements, processing elements, logic elements, lookup tables, etc., which may perform various functions under the control of one or more microprocessors or other control devices. Similarly, the software elements of the system may be implemented in any programming or scripting language such as: C, C++, C#, Object Notation (JSON), VBScript, Macromedia COLD FUSION, COBOL, The company's dynamic server pages, compilations, PHP, awk, Visual Basic, SQL stored procedures, PL / SQL, any shell scripts, and extensible markup language (XML), where various algorithms are implemented using any combination of data structures, objects, processes, routines, or other programming elements. In addition, it should be noted that the system can employ any number of conventional techniques for data transmission, signal transmission, data processing, network control, etc. Further, the system can be used to implement client-side scripting languages ​​such as VBScript, etc.) or prevent client-side scripting languages ​​such as VBScript, etc.) security issues.

[0115] The present invention describes systems and methods with reference to screenshots, block diagrams and flowchart illustrations of methods, apparatuses and computer program products according to various embodiments. It should be understood that each functional block in the block diagrams and flowchart illustrations, and the combination of functional blocks in the block diagrams and flowchart illustrations can be implemented by computer program instructions, respectively.

[0116] Therefore, the functional blocks of the block diagrams and flowcharts support a combination of means for performing the specified functions, a combination of steps for performing the specified functions, and program instruction means for performing the specified functions. It should also be understood that each functional block of the block diagrams and flowcharts, and the combination of functional blocks in the block diagrams and flowcharts can be implemented by a hardware-based special-purpose computer system that performs the specified functions or steps, or a suitable combination of special-purpose hardware and computer instructions. In addition, the diagrams and descriptions of the process flows can refer to the user Applications, web pages, websites, web forms, prompts, etc. Practitioners will appreciate that the illustrated steps described herein may be included in any number of configurations: including Applications, web pages, web forms, popups Use of applications, prompts, etc. It should also be understood that the multiple steps illustrated and described may be combined into a single web page and / or application, but has been expanded for simplicity. In other cases, steps illustrated and described as a single processing step may be separated into multiple pages and / or applications, but have been combined for simplicity.

[0117] Middleware may include any hardware and / or software that is appropriately configured to facilitate communication and / or processing transactions between disparate computing systems. Middleware components are commercially available and known in the art. Middleware may be implemented by commercially available hardware and / or software, by customized hardware and / or software components, or by a combination thereof. Middleware may reside in various configurations and may exist as an independent system, or may be a software component that resides on an internet server. Middleware may be configured to process transactions between the various components of an application server and any number of internal systems or external systems for any purpose disclosed herein. Inc. (Armonk, NY) MQTM (formerly MQSeries) is an example of a commercially available middleware product.An Enterprise Service Bus ("ESB") application is another example of middleware.

[0118] The computer discussed herein may provide a suitable website or other Internet-based graphical user interface accessible by a user. In one embodiment, The company's Internet Information Services (IIS), Transaction Server (MTS) services, and Database and operating system, Web server software, SQL Databases, and In addition, such as software, database, software, software, software, software, Components of software and the like can be used to provide a database management system that is compatible with Dynamic Data Objects (ADO). In one embodiment, Network Server and operating system, Database and PHP, Ruby, and / or Used in conjunction with programming languages.

[0119] For the sake of brevity, conventional data networking, application development, and other functional aspects of the system (as well as components of the various operating components of the system) may not be described in detail herein. In addition, the connecting lines shown in the various figures included herein are intended to represent exemplary functional relationships and / or physical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in an actual system.

[0120] In various embodiments, the methods described herein are implemented using various specific machines described herein. As will be immediately understood by those skilled in the art, the methods described herein can be implemented using the following specific machines and those machines developed hereinafter in any suitable combination. In addition, as will be apparent from this disclosure, the methods described herein can result in various transformations of certain articles of manufacture.

[0121] In various embodiments, the system and various components may be integrated with one or more intelligent digital assistant technologies. For example, exemplary intelligent digital assistant technologies may include Developed by the company system, GOOGLE developed by Alphabet, Inc. system, Company systems and / or similar digital assistive technologies. System, GOOGLE System and The systems can each offer cloud-based voice-activated services that can assist with tasks, entertainment, general information and more. Devices such as AMAZON AMAZON ECHO AMAZON and AMAZON TV, you can access system. System, GOOGLE System and The system can receive voice commands, activate other functions, control smart devices, and / or collect information via its voice activation technology. For example, the intelligent digital assistant technology can be used to interact with the following: music, email, text, phone, Q&A, home improvement information, smart home communication / activation, games, shopping, making to-do lists, setting alarms, streaming interactive podcasts, playing audiobooks, and providing weather, traffic and other real-time information such as news. GOOGLE and The system may also allow the user to access information about eligible transaction accounts linked to the online account on all digital assistant-enabled devices.

[0122] The various system components discussed herein may include one or more of the following: a host server or other computing system including a processor for processing digital data; a memory coupled to the processor for storing digital data; an input digitizer coupled to the processor for inputting digital data; an application stored in the memory and accessible by the processor for directing the processing of the digital data by the processor; a display device coupled to the processor and the memory for displaying information derived from the digital data processed by the processor; and a plurality of databases. The various databases used herein may include: client data; merchant data; financial institution data; and / or data useful for the operation of the system. As will be appreciated by those skilled in the art, a user computer may include an operating system (e.g., etc.) and various conventional supporting software and drivers typically associated with a computer.

[0123] The present system or any (one or more) part or (one or more) function thereof can be implemented using hardware, software or a combination thereof, and can be implemented in one or more computer systems or other processing systems. However, the operation performed by the embodiment can be referred to by terms such as matching or selecting, which are usually associated with the intellectual operation performed by a human operator. In most cases, in any operation described herein, this ability of a human operator is not necessary or undesirable. On the contrary, the operation can be machine operation, or any operation can be performed or enhanced by artificial intelligence (AI) or machine learning. AI can generally refer to the study of agents (e.g., machines, computer-based systems, etc.) that perceive the surrounding world, form plans, and make decisions to achieve their goals. The foundations of AI include mathematics, logic, philosophy, probability, linguistics, neuroscience, and decision theory. Many fields fall within the scope of AI, such as computer vision, robotics, machine learning, and natural language processing. Useful machines for performing each embodiment include general-purpose digital computers or similar devices.

[0124] In various embodiments, the embodiments are directed to one or more computer systems capable of performing the functions described herein. The computer system includes one or more processors. The processor is connected to a communication infrastructure (e.g., a communication bus, a crossover bar, a network, etc.). Various software implementations are described based on this exemplary computer system. After reading this description, it will be clear to those skilled in the relevant art how to use other computer systems and / or architectures to implement various implementations. The computer system may include a display interface that forwards graphics, text, and other data from the communication infrastructure (or from a frame buffer not shown) to display on a display unit.

[0125] The computer system also includes a main memory, such as a random access memory (RAM), and may also include a secondary memory. The secondary memory may include, for example, a hard disk drive, a solid state drive, and / or a removable storage drive. The removable storage drive reads from and / or writes to a removable storage unit in a well-known manner. As will be appreciated, a removable storage unit includes a computer-usable storage medium having computer software and / or data stored therein.

[0126] In various embodiments, auxiliary storage may include other similar devices for allowing computer programs or other instructions to be loaded into a computer system. Such devices may include, for example, removable storage units and interfaces. Such examples may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as erasable programmable read-only memories (EPROMs), programmable read-only memories (PROMs)) and associated sockets, or other removable storage units and interfaces that allow software and data to be transferred from the removable storage unit to the computer system.

[0127] The terms "computer program medium", "computer usable medium" and "computer readable medium" are used to generally refer to media such as removable storage drives and hard disks installed in hard drives, etc. These computer program products provide software to computer systems.

[0128] The computer system may also include a communication interface. The communication interface allows software and data to be transmitted between the computer system and an external device. Examples of such communication interfaces may include a modem, a network interface (such as an Ethernet card), a communication port, etc. The software and data transmitted via the communication interface are in the form of a signal, which may be electronic, electromagnetic, optical, or other signals that can be received by the communication interface. These signals are provided to the communication interface via a communication path (e.g., a channel). The channel carries signals and can be implemented using wires, cables, optical fibers, telephone lines, cellular links, radio frequency (RF) links, wireless and other communication channels.

[0129] As used herein, an "identifier" may be any suitable identifier that uniquely identifies an item. For example, an identifier may be a globally unique identifier ("GUID"). A GUID may be an identifier created and / or implemented under the Universal Unique Identifier standard. In addition, a GUID may be stored as a 128-bit value that may be displayed as 32 hexadecimal digits. An identifier may also include a major number and a minor number. Both the major number and the minor number may be 16-bit integers.

[0130] In various embodiments, the server may include an application server (e.g., POSTGRES PLUS ADVANCED In various embodiments, the server may include a web server (e.g., Apache, IIS, Network servers, System network server, or Running on the operating system virtual machines).

[0131] A network client includes any device or software that communicates via any network, such as, for example, any device or software discussed herein. A network client may include Internet browsing software installed in a computing unit or system to conduct online transactions and / or communications. These computing units or systems may take the form of a computer or group of computers, although other types of computing units or systems may be used, including personal computers, laptops, notebook computers, tablet computers, smart phones, cellular phones, personal digital assistants, servers, pool servers, mainframe computers, distributed computing clusters, kiosks, terminals, point of sale (POS) devices or terminals, televisions, or any other device capable of receiving data over a network. A network client may include an operating system (e.g., WINDOW operating system, operating system, operating system, operating system, etc.) and various conventional supporting software and drivers that are typically associated with a computer. The network client may also run INTERNET software, Software, GOOGLE CHROME TM software, software, or any of the countless software packages available for browsing the Internet.

[0132] As will be appreciated by those skilled in the art, a network client may or may not be in direct contact with a server (e.g., an application server, a network server, etc., as discussed herein). For example, a network client may access the services of a server through another server and / or hardware component, which may be directly or indirectly connected to an Internet server. For example, a network client may communicate with a server via a load balancer. In various embodiments, network client access is performed over a network or over the Internet through a commercially available web browser software package. In this regard, a network client may be in a residential or commercial environment that is capable of accessing a network or the Internet. A network client may implement security protocols, such as secure socket layer (SSL) and transport layer security (TLS). A network client may implement several application layer protocols, including HTTP, HTTPS, FTP, and SFTP.

[0133] The various system components may be coupled to the network independently, individually or collectively as appropriate via data links including, for example, connection to an Internet Service Provider (ISP) through a local loop that typically communicates with a standard modem, cable modem, DISH ISDN, digital subscriber line (DSL) or various wireless communication methods are used in combination. It is worth noting that the network can be implemented as other types of networks, such as interactive television (ITV) networks. In addition, the system contemplates the use, sale or distribution of any goods, services or information through any network with similar functions as described herein.

[0134] The system contemplates uses associated with network services, utility computing, pervasive and personalized computing, security and identity solutions, autonomic computing, cloud computing, commodity computing, mobile and wireless solutions, open source, biometrics, grid computing, and / or mesh computing.

[0135] Any communication, input, storage, database, or display discussed herein may be facilitated by a website having web pages. The term "web page" as used herein is not meant to limit the types of documents and applications that may be used to interact with a user. For example, in addition to standard HTML documents, a typical website may include various forms, Mini Programs, Programs, Active Server Pages (ASP), Common Gateway Interface scripts (CGI), Extensible Markup Language (XML), Dynamic HTML, Cascading Style Sheets (CSS), AJAX (Asynchronous JAVASCRIPT and XML) programs, helper applications, plug-ins, etc. The server may include a network service that receives a request from a network server, the request including a URL and an IP address (192.168.1.1). The network server retrieves the appropriate web page and sends the data of the web page or application to the IP address. A network service is an application that can interact with other applications through a communication method such as the Internet. Network services are typically based on standards or protocols such as XML, SOAP, AJAX, WSDL, and UDDI. Network service methods are well known in the art and are covered in many standard texts. For example, Representational State Transfer (REST) ​​or RESTful, network services can provide a way to achieve interoperability between applications.

[0136] The computing unit of the network client may also be equipped with an Internet browser that connects to the Internet or an intranet using standard dial-up, cable, DSL, or any other Internet protocol known in the art. Transactions originating from the network client may pass through a firewall to prevent unauthorized access by users from other networks. In addition, additional firewalls may be deployed between different components of the CMS to further enhance security.

[0137] Encryption can be performed by any technology that is now available or that may become available in the art - for example, Twofish, RSA, El Gamal, Schorr signatures, DSA, PGP, PKI, GPG (GnuPG), HPE format-preserving encryption (FPE), voltage, triple DES, Blowfish, AES, MD5, HMAC, IDEA, RC6, and symmetric and asymmetric cryptographic systems. The system and method can also be combined with the SHA family of cryptographic methods, elliptic curve cryptography (e.g., ECC, ECDH, ECDSA, etc.), and / or other post-quantum cryptographic algorithms being developed.

[0138] The firewall may include any hardware and / or software that is appropriately configured to protect CMS components and / or enterprise computing resources from users of other networks. In addition, the firewall may be configured to limit or restrict access to various systems and components behind the firewall of network clients connected through a network server. The firewall may reside in different configurations, including stateful inspection, proxy-based, access control lists, and packet filtering. The firewall may be integrated into a network server or any other CMS component, or may reside as a separate entity in addition. The firewall may implement network address translation ("NAT") and / or network address port translation ("NAPT"). The firewall may adapt to various tunneling protocols to facilitate secure communications, such as those used in virtual private networks. The firewall may implement a demilitarized zone ("DMZ") to facilitate communication with a public network (such as the Internet). The firewall may be integrated into an Internet server or any other application server component as software, reside in another computing device, or take the form of an independent hardware component.

[0139] Any database discussed herein may include relational, hierarchical, graph, blockchain, object-oriented structures, and / or any other database configuration. Any database may also include a flat file structure, where data may be stored in a single file in the form of rows and columns, with no structure for indexing, and without structural relationships between records. For example, a flat file structure may include a delimited text file, a CSV (comma separated value) file, and / or any other suitable flat file structure. Common database products that may be used to implement a database include (Armonk, NY) Available from Corporation (Redwood Shores, CA) various database products, MICROSOFT from Corporation(Redmond,Washington) or MICROSOFT SQL MySQL AB (Uppsala, Sweden) Redis, Apache The company's MapR-DB, or any other suitable database product. In addition, any database can be organized in any suitable manner, for example, as a data table or a lookup table. Each record can be a single file, a series of files, a linked series of data fields, or any other data structure.

[0140] As used herein, big data may refer to partially or fully structured, semi-structured, or unstructured data sets that include millions of rows and hundreds of thousands of columns. For example, a big data set may be compiled from purchase transaction history over a period of time, from web registrations, from social media, from records of charges (ROC), from summaries of charges (SOC), from internal data, or from other suitable sources. A big data set may be compiled without descriptive metadata such as column types, counts, percentiles, or other explanatory data points.

[0141] The association of certain data can be achieved by any desired data association technique (such as those known or practiced in the art). For example, the association can be done manually or automatically. Automatic association techniques can include, for example, database searches, database merges, GREP, AGREP, SQL, using key fields in tables to speed up searches, sequential searches through all tables and files, sorting records in files according to a known order to simplify searches, etc. The association step can be accomplished by a database merge function, for example, using "key fields" in a pre-selected database or data sector. Various database tuning steps are envisioned to optimize database performance. For example, frequently used files (such as indexes) can be placed on a separate file system to reduce input / output ("I / O") bottlenecks.

[0142] More specifically, "key fields" divide the database according to the high-level categories of objects defined by the key fields. For example, certain types of data can be designated as key fields in multiple related data tables, and the data tables can then be linked based on the type of data in the key fields. The data corresponding to the key fields in each of the linked data tables is preferably the same or of the same type. However, data tables with similar but not identical data in the key fields can also be linked by using, for example, AGREP. According to one embodiment, any suitable data storage technology can be used to store data that does not have a standard format. Any suitable technique may be used to store the data sets, including, for example, storing a single file using the ISO / IEC 7816-4 file structure; implementing a domain, thereby selecting a dedicated file that exposes one or more elementary files containing one or more data sets; utilizing data sets stored in a single file using a hierarchical archiving system; data sets stored as records in a single file (including compressed, SQL accessible, hashed via one or more keys, numeric, alphabetically arranged by first tuple, etc.); data stored as binary large objects (BLOBs); data stored as ungrouped data elements encoded using ISO / IEC7816-6 data elements; data stored as ungrouped data elements encoded using ISO / IEC Abstract Syntax Notation (ASN.1), such as in ISO / IEC 8824 and 8825; other proprietary techniques, which may include fractal compression methods, image compression methods, etc.

[0143] In various embodiments, the ability to store multiple information in different formats is facilitated by storing information as BLOBs. Therefore, any binary information can be stored in a storage space associated with a data set. As discussed above, binary information can be stored in association with a system or outside the system but attached to the system. The BLOB method can use fixed storage allocation, circular queue technology, or best practices for memory management (e.g., paging memory, least recently used, etc.) to store data sets as ungrouped data elements formatted as binary blocks via fixed memory offsets. By using the BLOB method, the ability to store various data sets with different formats facilitates storing data in a database or storing data associated with a system by multiple and unrelated owners of the data set. For example, a first data set that can be stored can be provided by a first party, a second data set that can be stored can be provided by an unrelated second party, and a third data set that can be stored can be provided by a third party that is unrelated to the first party and the second party. Each of these three exemplary data sets can contain different information stored using different data storage formats and / or technologies. In addition, each data set can contain a subset of data that may also be different from other subsets.

[0144] As stated above, in various embodiments, data can be stored without considering the common format. However, when providing for manipulating data in a database or system, a data set (e.g., BLOB) can be annotated in a standard manner. The annotation can include a short header, a footer, or other appropriate indicators associated with each data set, and each data set is configured to convey information useful for managing various data sets. For example, the annotation may be referred to as a "conditional header," "header," "footer," or "status" in this article, and may include an indication of the state of the data set or may include an identifier associated with a specific publisher or owner of the data. In one example, the first three bytes of each data set BLOB may be configured or can be configured to indicate the state of the specific data set; for example, loaded, initialized, ready, blocked, removable, or deleted. The subsequent bytes of the data may be used to indicate the identity of, for example, the issuer, user, transaction / member account identifier, etc. Each of these conditional annotations is further discussed herein.

[0145] Dataset annotations may also be used for other types of state information and for various other purposes. For example, dataset annotations may include security information that establishes access levels. For example, access levels may be configured to allow only certain individuals, levels of employees, companies, or other entities to access a dataset, or to allow access to specific datasets based on transactions, merchants, publishers, users, etc. In addition, security information may only restrict / allow certain actions, such as accessing, modifying, and / or deleting a dataset. In one example, a dataset annotation indicates that only the dataset owner or user is allowed to delete the dataset, various identified users may be allowed to access the dataset for reading, and other users may be completely excluded from accessing the dataset. However, other access restriction parameters may also be used, allowing various entities to access datasets with various appropriate permission levels.

[0146] The data including the header or footer may be received by a separate interactive device configured to add, delete, modify or supplement data based on the header or footer. Thus, in one embodiment, the header or footer is not stored on the transaction device along with the associated issuer owned data, but appropriate action may be taken by providing the user at the separate device with appropriate options for the action to be taken. The system may contemplate a data storage arrangement in which a header or footer of data, or a history of headers or footers, is stored on the system, device or transaction tool in association with appropriate data.

[0147] Those skilled in the art will also understand that for security reasons, any database, system, device, server, or other component of the system may include any combination thereof at a single location or multiple locations, with each database or system including any of a variety of suitable security features (such as firewalls, access codes, encryption, decryption, compression, decompression, etc.).

[0148] Practitioners will also appreciate that there are many methods for displaying data within a browser-based document. Data may be presented as standard text or in a fixed list, scrollable list, drop-down list, editable text field, fixed text field, pop-up window, etc. Likewise, there are many methods for modifying data in a web page, such as, for example, entering free text using a keyboard, selecting menu items, check boxes, option boxes, etc.

[0149] The data may be big data processed by a distributed computing cluster. The distributed computing cluster may be, for example, A software cluster configured to process and store large data sets, where some nodes include a distributed storage system and some nodes include a distributed processing system. In this regard, the distributed computing cluster can be configured to support the Apache Software Foundation as specified at www.hadoop.apache.org / docs. Software Distributed File System (HDFS).

[0150] As used herein, the term "network" includes any cloud, cloud computing system, or electronic communication system or method that combines hardware and / or software components. Communications between parties may be accomplished through any suitable communication channel, such as, for example, a telephone network, an extranet, an intranet, the Internet, a point of interaction device (point of sale device, a personal digital assistant (e.g., equipment, devices), cellular phones, kiosks, etc.), online communications, satellite communications, offline communications, wireless communications, repeater communications, local area networks (LANs), wide area networks (WANs), virtual private networks (VPNs), networking or linking devices, keyboards, mice, and / or any suitable form of communication or data entry. In addition, although the system is often described herein as being implemented using the TCP / IP communication protocol, the system may also use IPX, The invention can be implemented using a protocol such as TCP, IP-6, NetBIOS, OSI, any tunneling protocol (e.g., IPsec, SSH, etc.), or any number of existing or future protocols. If the network is in the nature of a public network, such as the Internet, it may be advantageous to assume that the network is insecure and open to eavesdroppers. Specific information regarding protocols, standards, and application software used in conjunction with the Internet is generally known to those skilled in the art and does not need to be described in detail here.

[0151] "Cloud" or "cloud computing" includes models for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. Cloud computing can include location-independent computing that delivers resources, software, and data on demand to computers and other devices through shared servers.

[0152] As used herein, "transmission" may include sending electronic data from one system component to another system component via a network connection. In addition, as used herein, "data" may include information in digital or any other form including commands, queries, files, data for storage, etc.

[0153] Any database discussed herein may include a distributed ledger maintained by multiple computing devices (e.g., nodes) on a peer-to-peer network. Each computing device maintains a copy and / or partial copy of the distributed ledger, and communicates with one or more other computing devices in the network to verify data and write data to the distributed ledger. The distributed ledger may use features and functions of blockchain technology, including, for example, consensus-based verification, immutability, and encrypted linked data blocks. The blockchain may include a ledger containing interconnected blocks of data. The blockchain may provide enhanced security because each block can save separate transactions and the results of any blockchain executable file. Each block may be linked to the previous block and may include a timestamp. Blocks may be linked because each block may include a hash of the previous block in the blockchain. The linked blocks form a chain in which only one successor block is allowed to link to another predecessor block of a single chain. In the case of establishing a divergent chain from a previously unified blockchain, a fork may occur, although typically only one divergent chain will be maintained as a consensus chain. In various embodiments, the blockchain may implement a smart contract that implements a data workflow in a decentralized manner. The system may also include applications deployed on user devices, such as, for example, computers, tablets, smartphones, Internet of Things devices ("IoT" devices), etc. The applications may communicate with the blockchain (e.g., directly or via a blockchain node) to transfer and retrieve data. In various embodiments, a management organization or consortium may control access to data stored on the blockchain. Registration with (one or more) management organizations may allow participation in the blockchain network.

[0154] Data transmissions performed through a blockchain-based system may be propagated to connected peers within the blockchain network for a duration that may be determined by the block creation time of the particular blockchain technology implemented. On a network based on On the Fabric 1.0 platform, the duration is driven by the specific consensus algorithm selected and can be executed in seconds. In this regard, the propagation time in the system can be improved compared to existing systems, and the implementation cost and time to market can also be greatly reduced. The system also provides higher security, at least in part due to the immutability of the data stored in the blockchain, reducing the possibility of tampering with various data inputs and outputs. In addition, the system can also provide higher data security by performing cryptographic processing on the data before storing it on the blockchain. Therefore, by transmitting, storing, and accessing data using the system described in this article, the security of the data is improved, which reduces the risk of computers or networks being compromised.

[0155] In various embodiments, the system can also reduce database synchronization errors by providing a common data structure, thereby at least partially improving the integrity of stored data. The system also provides higher reliability and fault tolerance than traditional databases (e.g., relational databases, distributed databases, etc.) because each node operates with a complete copy of the stored data, thereby at least partially reducing downtime due to local network outages and hardware failures. The system can also improve the reliability of data transmission in a network environment with reliable and unreliable peers because each node broadcasts messages to all connected peers, and because each block includes a link to the previous block, the node can quickly detect the missing block and propagate the request for the missing block to other nodes in the blockchain network.

[0156] The specific blockchain implementations described herein provide improvements over conventional techniques by using a decentralized database and an improved processing environment. In particular, the blockchain implementations improve computer performance by, for example, utilizing decentralized resources (e.g., lower latency). Distributed computing resources improve computer performance by, for example, reducing processing time. In addition, distributed computing resources improve computer performance by improving security using, for example, cryptographic protocols.

[0157] Any communication, transmission, and / or channel discussed herein may include any system or method for delivering content (e.g., data, information, metadata, etc.) and / or the content itself. The content may be presented in any form or medium, and in various embodiments, the content may be delivered electronically and / or may be capable of being presented electronically. For example, a channel may include a website, a mobile application, or a device (e.g., GOOGLE CHROMECAST TM , etc.) Uniform Resource Locators ("URLs"), documents (e.g., Word or ExcelTM , Portable Document Format (PDF) documents, etc.), "electronic books", "electronic magazines", applications or micro-applications (as described herein), short message service (SMS) or other types of text messages, emails, information, In various embodiments, the distribution channel may include at least one of a merchant website, a social media website, an affiliated or partner website, an external supplier, a mobile device communication, a social media network, and / or a location-based service. The distribution channel may include at least one of a merchant website, a social media website, an affiliated or partner website, an external supplier, and a mobile device communication. Examples of social media websites include

[0158]

[0159] etc. Examples of affiliated or partner websites include AMERICAN wait.

Claims

1. A method, comprising: receiving, by a processor, an image of a human being from a sensor; receiving, by the processor, a location of a bounding box on the image, wherein the bounding box contains pixel data for the human in the image; obtaining, by the processor, bounding box data from within the bounding box; and A pose of the human is determined, by the processor, based on the bounding box data.

2. The method according to claim 1, further comprising: A neural network is trained, by the processor, to predict a location of the bounding box on the image.

3. The method according to claim 1, further comprising: The neural network is trained by the processor using the pixel data of the human, the thermal data of the human, and the environmental data.

4. The method according to claim 1, further comprising: The processor adjusts the algorithm of the neural network based on the environmental data.

5. The method according to claim 1, further comprising: The processor adjusts an algorithm of the neural network based on environmental data, wherein the environmental data includes at least one of the following: ambient temperature, room temperature, floor plan, non-human thermal objects, gender of the human, age of the human, height of the sensor, clothing of the human, or body weight of the human.

6. The method according to claim 1, wherein: The sensor acquires thermal data about the human being.

7. The method according to claim 1, wherein: A user indicates the location of the bounding box on the image.

8. The method according to claim 1, wherein: The posture is determined for frames in an image captured by the sensor.

9. The method according to claim 1, wherein: Determining the posture further includes determining an aggregated posture over a period of time across a plurality of frames.

10. The method according to claim 1, further comprising: A fall is determined by the processor based on the aggregated posture changing from at least one of a standing posture or a sitting posture to a lying posture and the lying posture persisting for a certain amount of time.

11. The method according to claim 1, further comprising: Distinguishing features are extracted by the processor from a plurality of frames of the image using pattern recognition.

12. The method according to claim 1, further comprising: The resolution of the image is limited, by the processor, based on at least one of: privacy concerns, power consumption of the sensor, cost of the pixel data, bandwidth of the pixel data, computational cost, or computational bandwidth.

13. The method according to claim 1, further comprising: The processor labels a pose of the human in the image.

14. The method according to claim 1, wherein: The image is part of a video clip of the human being.

15. The method according to claim 1, wherein: Acquiring the bounding box data includes acquiring the bounding box data at least one of over time or during an initial calibration session.

16. The method according to claim 1, further comprising: Distinguishing features from the thermal signature pattern of the top of the human head are analyzed by the processor to determine the posture of the human.

17. The method according to claim 1, wherein: The posture includes at least one of: sitting, standing, lying, exercising, dancing, running, or eating.

18. The method according to claim 1, further comprising: determining, by the processor, a temperature of the human in the space based on infrared energy data regarding infrared (IR) energy from the human; The processor determines the position coordinates of the human in the space; Comparing the position coordinates of the human being with the position coordinates of a fixed object by the sensor system; as well as In response to the temperature of the object being within a range and in response to the position coordinates of the human being being different from the position coordinates of the fixture, it is determined by the sensor system that the human being is a human. The method of claim 1 further comprising determining, by the processor, a trajectory of the human based on changes in temperature in the pixel data, wherein the temperature is projected onto a grid of pixels.

19. An article of manufacture comprising a non-transitory tangible computer-readable storage medium having instructions stored thereon that, in response to execution by a computer-based system, cause the computer-based system to perform operations comprising: receiving, by the processor, an image of a human being from a sensor; The processor receives a position of a bounding box on the image, wherein: The bounding box contains pixel data of the human in the image; acquiring, by the processor, bounding box data from within the bounding box; as well as A pose of the human is determined, by the processor, based on the bounding box data.

20. A system, comprising: processor; as well as a tangible non-transitory memory configured to communicate with the processor, The tangible, non-transitory memory has instructions stored thereon that, in response to execution by the processor, cause the processor to perform operations including: receiving, by the processor, an image of a human being from a sensor; receiving, by the processor, a location of a bounding box on the image, wherein the bounding box contains pixel data for the human in the image; obtaining, by the processor, bounding box data from within the bounding box; and A pose of the human is determined, by the processor, based on the bounding box data.

Images