Method for real-time integrated "t+2D" DWT video compression and low-level video analysis within an intelligent camera optionally integrated into an end-to-end AAL system

Abstract

1. A method and apparatus for processing video data, comprising: using a camera with a computing device having memory storage and power to capture and store video data; and simultaneously compressing and analyzing the video data using a low level analysis of contours of stationary and moving objects in the video data, wherein the compression may be wavelet-based decomposition; and wherein the analysis of the video data compares real-time semantic human actions in the video data, thereby identifying specific human actions in the video data; and wherein the camera is wirelessly connected to a base station and capable of communicating over a two-way full duplex connection.

Description

[Technical field]

[0001] Inventor: Mr. Radmilo Bozinovic; Mr. Isaak E. Van Kempen Agent File Number: 0595PCT

[0002] For the United States only, this application is a continuation-in-part (CIP) of U.S. Patent Application No. 16,653,902, filed October 15, 2019, and claims the benefit of U.S. Provisional Patent Application No. 62,746,483, filed October 16, 2018, which applications are incorporated by reference in their entireties.

[0003] TECHNICAL FIELD OF THEINVENTION The present invention relates to improved techniques for integrating video compression and low-level video analysis. [Background technology]

[0004] Huge amounts of video data are being generated daily from a variety of devices on an unprecedented scale, with or without the need to interpret this information in real time. Not only is the volume of data growing rapidly, but so is its variety (range of data types, resolution, and sources) and velocity (speed of acquisition and transmission), which are in keeping with Gartner's "3Vs of Big Data". To facilitate useful processing of such data, some level of real-time automated video analytics has become widespread, but there are still significant gaps and potential for growth in areas such as surveillance, automotive, robotics, and healthcare / welfare. Although seemingly diverse, all these fields are often integrated into broader emerging paradigms such as Ambient Intelligence (AmI), Ambient Assisted Living (AAS), Aging in Place (AiP), and Smart Surveillance - they share some key challenges and requirements, and are likely to utilize the same underlying platforms.

[0005] For a variety of reasons, ranging from practical to legal, it is widely accepted that most automated surveillance / monitoring must be corroborated by original (possibly compressed) video footage. However, effective real-time compression of an exponentially growing variety of videos remains problematic, since the prevailing standard codecs (H.26x) were designed with different (broadcast) requirements and cannot be optimally integrated in this context, despite progress in understanding the problem (e.g. attempts to amend the IEEE 1857 / AVS standard). Moreover, it is well-known that video analysis in the compressed domain is inherently difficult, and similarly, subsequent video analysis downstream in the processing chain (e.g. "in the cloud") must endure expensive and impractical codec cycles. Together with the ongoing pressure to reduce the weight and power consumption of capture devices, this defines the current main challenge to the widespread adoption of automated real-time video analytics. As summarized in “Human Activity Analysis: A Review” in Computing Surveys 43(3), “This computational gap must be overcome in order to apply activity recognition methods to real-world applications such as surveillance systems, human-computer interfaces, intelligent robots, and autonomous vehicles.” One way to alleviate this bottleneck is therefore to optimally map the leading video analytics techniques (which are generally well understood by now in terms of design and tradeoffs) to ongoing improvements in common hardware (e.g., multicore CPUs, GPGPUs, specialized hardware), and specifically – to tightly integrate video analytics and compression as early as possible in the processing chain.

[0006] Such an approach is further supported by two challenges: privacy and security. That is, it is recommended to exclude the full video, regardless of compression or encryption, from most of the public networks (not only in the workflow processing chain) whenever possible, and a possible and common analytical goal is known in advance. All this essentially supports the need to try and maintain a unified process of compression and analysis at the "head-end" (i.e., intelligent camera), with bifurcated outputs: the full compressed video (for a time limited by available memory) stored locally on the camera, and the essential information (such as pre-processed footage for optimal recognition, indexing and retrieval) passed on low bandwidth. Rather than simply focusing on mathematical redundancy to achieve data reduction, here we remove non-essential information, allowing a kind of "semantic compression" that achieves both goals simultaneously.

[0007] Finally, we would like to emphasize that the benefits of such an approach are not limited to applications that require real-time video analysis, but rather can increase the usefulness of big data as a whole. This is determined by a confluence of factors: a) the sheer volume and growth of fresh video material generated every day, b) the need for video indexing and retrieval techniques to effectively manage, store and utilize such a volume and variety of material, and c) the fact that applying a level of video indexing at the source, together with compression and analysis, can significantly simplify the offline retrieval task. From the book "Visual Indexing and Retrieval" in Springer Briefs, 2012: "Even in the field of social activities such as video surveillance and security, thousands of terabytes of video content are generated, which have specific problems to be addressed. ... Thus, visual information indexing and retrieval has attracted a lot of research efforts since the early 90s. Nowadays, however, the research directions are wide open due to the size of large databases, the complexity of the visual interpretation task, and the well-known "semantic gap" between the (semantic) concepts that users are looking for in an image and the digital representation of the (semantic) visual content. " Therefore, integrating some degree of visual analytics at the point of video capture and compression can reduce this semantic gap while at the same time alleviating the challenge of the "three V's" information bottleneck. At the same time, such an approach effectively implements the "edge computing" paradigm, thus justifying a concise summary from an authoritative research paper, which states that "the rationale for edge computing is to perform computing in the vicinity of the data source" (Shi, et al.: Edge Computing Vision and Challenges, IEEE IoT Journal, Vol.3, No.5, Oct.2016), and fulfills one of the key advantages of "data abstraction" and "data should be preprocessed at the gateway level, such as removing noise and low quality, detecting events, and protecting privacy."

[0008] The present invention describes a method for optimally integrating two previously separate and sequential operations: video compression and low-level analysis. The method is incorporated into a camera subsystem and designed to facilitate some real-time semantic human activity recognition and communication with remote mobile device users. In one embodiment (presented as a primary illustration of the applicability of the invention), these users are remote family caregivers or healthcare providers who are then part of a broader AAL system enabling AiP for the monitored elderly. However, the invention is not limited to such a specific application and other embodiments can be envisioned, such as smart monitoring for security applications, where Events of Interest (EOI, see component 500 in the invention disclosure below) can alternatively include detection of intrusion / trespass (unauthorized or other suspicious personnel in a restricted area), tailing (illegal follow-through at a controlled access point), suspicious object (unattended items deposited in a sensitive area beyond predefined limits), loitering (persons seen in a controlled area for an extended period of time), and possibly other selected predefined anomalous activities (e.g. violent physical contact, person lying on the ground).

[0009] As alluded to above, the present invention consequently embodies aspects of the edge computing paradigm, along with elements of edge analytics and cloudlet capabilities (see, e.g., M. Satyanarayanan: The Emergence of Edge Computing, IEEE Computer, 2017).

[0010] The present invention introduces such improvements. In its preferred embodiment, the invention has several aspects or aspects which can be used independently, but are preferably employed together to optimize their advantages. All of the foregoing operating principles and advantages of the present invention will be more fully understood when considered in conjunction with the following detailed description taken in conjunction with the drawings. Summary of the Invention

[0011] The present invention provides a method and apparatus for integrating video compression and low-level video analysis to determine people's specific behavior. Such an apparatus can operate autonomously in a wide range of applications that require real-time automated analysis of human behavior backed by original footage and with privacy and security in mind. In one (preferred) embodiment, it is part of a broader system of Ambient Assisted Living (AAL), acting as a client module for monitoring the health and behavior of Ageing-in-Place (AiP), and coupled with a network server. The purpose of such a network server is to host back-end modules (big blocks) including Authentication, Authorization, and Accounting (AAA) servers, applicant applications, and big data processing, thereby providing a reliable, scalable, highly available, and easy to monitor and manage deployment. In this embodiment, it selects and employs modules corresponding to the applicant's basic data flows, which include user registration, authentication, authorization, and accounting, facilitates secure uploading, management, and archiving of user data files, and performs big data processing on the uploaded data files according to the business requirements of the remotely connected users.

[0012] The present invention provides an apparatus and method for processing video data, comprising: (a) using a camera having a computing device with memory storage and power source to capture and store video data; (b) compressing and simultaneously analyzing said video data using a low level analysis of contours of stationary and moving objects in the video data, where, without limitation, said compression may be wavelet based, said analysis may use or employ contour extraction and edge detection, and said compression may integrate or employ said analysis within the compression workflow.

[0013] Further, the camera can be wirelessly connected to the base station and can communicate via a 2-way full duplex connection, and the camera can transmit and stream outline video data and ternary gray level images, flat backgrounds, outlines of stationary objects, or outlines of moving objects, and the learning module can be used or learned to recognize units and events and trends of said units and events over time, and the learning module can be used or learned to recognize long-term changes in daily life, behaviors such as gait, bending, walking, rise time, etc., or long-term changes in audio data, and data from sensors that are binary, discrete value, or continuous value, or long-term changes in the surrounding environment, lighting, doors, windows, and room furnishings.

[0014] Additionally, the analysis of the video data can compare real-time semantic human activity within the video data, thereby enabling the method to identify specific human activity within the video data. The video data is a video sequence of a two-dimensional N×K array of numbers, and low level analysis is used to generate salient contour features of moving and stationary scene objects and humans, and the method can provide automated scene analysis of the video sequence, and the method can also use or employ real-time compression and automated visual analysis of the source video signal of the two-dimensional array of numbers.

[0015] As shown in Figure 2, the data sequence overview is as follows: (1) a user and its associated base station (BS) are registered with the AAA server. (2) the BS sends the processed video data files to the back-end processing. (3) each file upload request is authenticated by the AAA server. (4) the user data is processed by the big data engine (in a preferred embodiment, the big data engine performs an ETL (Extract-Transform-Load) process on the user data to prepare the data in the format required by the business logic so that it can be displayed on the user dashboard). (5) the dashboard presents the processed data via data visualization software.

[0016] Applicant's network server application may have the following main functions: (1) interface with the AAA server to register and manage users and base stations (clients); (2) upload, store, and manage data files from registered base stations; (3) allow registered users to log in and view data files / statistics / metrics for mapped base stations; (4) provide the following management functions: add new users and register them with the AAA server; add new clients (base stations) and register them with the AAA server; provide a way to map users to base stations (one-to-one, one-to-many, and many-to-many); provide the ability to view the upload directory; keep a log of all server operations; and define the retention period for uploaded files. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 shows a client subsystem, which consists of a camera module that captures schematic or complete video data and is connected to a base station. [Diagram 2] FIG. 2 shows a preferred embodiment of the data sequence. [Diagram 3]FIG. 3 is a diagram showing the Initial Deployment Proposed Technology Stack. [Figure 4] Figure 4 shows the network server deployment diagram, and the terms and acronyms are: AWS-Amazon Web Services VPC-Virtual Private Cloud EC2-Elastic Compute Cloud-AWS Server EFS-Elastic File Storage RDS-Relational Database Service ELB-Elastic Load Balancer [Diagram 5] Figure 5 is a graph showing the Basic Deployment Facts. [Figure 6] Figure 6 shows an overview of a camera module where data from an imaging sensor is first temporally and spatially decomposed and then processed to perform simultaneous compression and low-level video analysis. [Figure 7] FIG. 7 shows an overview of a preferred embodiment of the present invention, comprising a camera module, a base station, a network server and a decoder. [Figure 8] FIG. 8 is a schematic diagram of a computing system connecting to a network server. [Figure 9] FIG. 9 is a schematic diagram of a computing system with a communications network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] First embodiment The present invention describes a camera device ("camera module" -CM) capable of emitting / streaming an outline video (OV: here meaning a ternary gray level image: / flat background / contours of stationary objects / contours of dynamic objects / each with different gray levels) and simultaneously recording a compressed version of the full video (FV) in its available internal memory (ring buffer). This is achieved by a method that optimally integrates ("conceptually in one operation") two operations related to video processing that have been performed separately and sequentially up to now: compression and low-level analysis (specifically contour extraction / edge detection). In particular, elements of a known class of video compression methods (wavelet-based decomposition, aka DWT), usually used only for the purpose of encoding / compressing video signals, are used simultaneously in a video analysis and recognition workflow.

[0019] The camera module (CM) may be multiple, audio and video input devices, and data collection devices. The independent quality (and practicality) of this embodiment comes from the fact that the outline video is human perceptible and meaningful (apart from being an input and the first stage of further possible processing, it can be organically blended). Furthermore, it is recognized that the usefulness of providing only the outline video to the user is practically justified in certain applications, especially those where privacy and security considerations are important. Furthermore, nothing in this description conceptually prevents the camera module from streaming full video instead of the outline video.

[0020] Second embodiment There can be multiple or at least one Camera Module (CM) as described above, equipped with a client subsystem, communicating in a two-way (full duplex) connection and coupled with a Base Station (BS). In the main direction flow (CM → BS), the usual content is outline video, which is used by the Base Station as a first step in extensive real-time semantic human activity recognition. When a predefined Event of Interest (EOI; Event of Interest, e.g. fall detection) is detected, a reverse BS → CM (singular) communication occurs, signaling the CM to switch from OV to FV transmission. The CM ← → BS connection can use wireless (Wi Fi LAN) or wired or other wireless communication connections.

[0021] There is a wide range of human activity recognition options here (some of which are described for illustrative purposes in the provisional application, with fall detection being one of the most substantial and of most interest in the third embodiment).

[0022] Third embodiment The overall Ambient Assisted Living (AAS) system intended to promote Aging in Place (AiP) comprises a network server (NS) and at least one or more mobile viewers (MVs) in addition to the client subsystem integrating the CM and BS mentioned above, where the actual real-time semantic human activities would be mainly the “Activities of daily living” (ADS) at home of elderly subjects in addition to the detection of specific EOIs (e.g. life-threatening human falls) mentioned in the second embodiment.

[0023] The network server provides basic authentication, authorization and accounting (AAA) functions to registered users and base stations, hosts data analysis modules and presents content to these users. Mobile viewers are applications hosted on remotely connected (broadband or cellular network) mobile devices, available to registered users (e.g. family members or healthcare providers) and act as caregivers in a broader system enabling safe Aging in Place (AiP) for monitored elderly people. These mobile viewers provide access to live outline video (full video if an Event of Interest (EOI) is detected) as well as an information dashboard of statistics / trends. If the base station detects an EOI, it immediately notifies all registered users' mobile viewers and full video streaming is available. Even in a single network server (NS), there can be multiple base stations, camera modules and mobile viewers, and the functionality of the NS can be elaborated in different ways, such as simple or complex configurations. Beyond basic ADLs and EOI, there is useful information sent from the BS to the NS for long-term ("big data") baseline, trend, and change analysis (e.g., walking, gait, and hip bending of the target resident or patient).

[0024]

[0100] An imaging sensor that produces 2D raster images (color or grayscale) continuously at a certain frequency (in practice 10-12 fps or more).

[0025]

[0200] The image processor (functionally part of the camera module and physically contained in the camera housing, implemented in one embodiment as a general-purpose CPU / GPU loaded with memory and appropriate software) performs a "t+2D" spatio-temporal decomposition (alternative: DWT-based binary decomposition / compression) cumulatively by performing the following two decompositions separately in time and space in sequence. Such methods are commonly known and available to those skilled in the art: (See, for example, Jens-Rainer Ohm, Mihaela van der Schaar, John W. Woods, "Interframe wavelet coding- motion picture representation for universal scalability", Signal Processing, Image Communication 19 (2004), pp. 877-908: in particular, Figure 1 therein gives a clear illustration of the spatio-temporal wavelet decomposition using T=3 levels of the temporal wavelet tree; G. Karlsson, M. Vetterli, "Subband coding of video signals for packet switched networks", Proc. Visual Commun. Image Process, 845 (1987), pp. 446-456).

[0026]

[0220] Based on the stored sequence of previous images / frames, the unit first performs the temporal part of the "t+2D" decomposition over T levels: at each level it splits into low-pass (L) and high-pass (H) components, and then it processes recursively over the L components of the previous levels. In one (preferred) embodiment, the wavelet decomposition along the time axis can use a Haar basis, although other wavelet schemes can also be used for this unit. In this preferred case, the T-level decomposition involves a Haar basis of length 2 T A time block of frames is required. In practice, the value of T should not be high (e.g., 1-3). Regardless of the number of levels, we define the final (coarsest) decomposition set as L * and H *(each is a 2D array). H * is then (optionally) subjected to some thresholding, leaving a binary image of the active contour. In alternative embodiments, this temporal hierarchical decomposition can be further exploited, optionally performing a similar thresholding on the earlier H components, to provide contours of faster moving objects / parts, thereby providing a measure of temporal flexibility and scalability.

[0027]

[0240] Said L * A unit that takes the array / image as input and further performs a standard multi-level spatial DWT based binomial decomposition / compression into 4 bands per spatial level: this creates, for each level, the standard LL, LH, HL, HH bands (i.e. 2D arrays), the last three bands considered as "high frequency", and the first "low frequency" band, and at the final (coarsest) level, these bands are ... * , L.H. * , H.L. * , H.H. * In practice (or in one embodiment), the number of spatial levels will of course also depend on the initial resolution of the image, but should be kept to preserve the basic features of the desired object (e.g., the basic contour and pose of the observed human). As with 220, alternative embodiments can further exploit this spatial hierarchical decomposition, optionally performing a similar thresholding operation on the previous H components to provide coarser contours of static objects, thereby providing a measure of spatial flexibility and scalability.

[0028]

[0300] In the conceptually separate merge module, the three high frequency bands are first (optionally) thresholded, then these three high frequency bands are connected (superimposed) into one contour image, leaving a binary image of static contours that is later further transmitted to the base station 500, and finally the H *The resulting contour view 2D array (in one embodiment pursued here) contains ternary pixels, simultaneously displaying stationary object contours (one gray-white level) and moving object contours (another gray-white level) (assumed without loss of generality, on a black background). All operations described herein may be generalized to the alternative multi-level embodiments described in 220 and 240 in a manner obvious to one skilled in the art. Similarly, in another alternative embodiment, the contour view pixels need not be limited to three values, but rather a range of "gray" values ​​are added to indicate different motion speed levels.

[0029]

[0400] A circular (ring) memory buffer in which the continuously updated / overwritten compressed full image (of a predefined duration Δt) is stored by using any standard techniques for quantization and entropy coding of the temporal and spatial L and H bands, hierarchically generated in the above-mentioned units 200 to 240.

[0030]

[0500] A base station receives (via wired or wireless communication - in one embodiment through a WiFi port) a sequence of contour "ternary" images and performs a continuous automatic visual analysis with the aim of generative decision making, i.e. a base station that continuously detects instances of daily living events (EDLs), which may belong to several levels or types, but always includes the highest, alarm-generating level of EOI (event of interest; mainly AiP application: human falling). When an EOI instance is detected, a one-bit signal is sent to the camera processor 200, which switches from sending a contour view / video to a full (compressed) view. At the same time, such a full view compressed video is no longer recorded in the ring buffer 400, making it possible to save the last Δt of the full video preceding the EOI (to be accessed later via a separate interface via unit 900). In the preferred embodiment, the primary input of this module is data abstracted from video input, however its reception and integration capabilities are not limited to that domain and it should be straightforward for one skilled in the art to generalize to include other data inputs (including audio and miscellaneous binary, discrete and continuous value sensors - wearable, device monitoring or ambient).

[0031]

[0510] In one embodiment, a higher level video analytics module is based on a widely accepted (see, for example, the 2011 Springer book "Visual Analysis of Humans") classification of four main types of activity: kinetics (human and object movement), proxemics (human proximity to humans and objects), haptics (human contact with objects), and chronemics (changes over time). The approach is outlined as follows and is loosely based on a Hidden Markov Model network (HMM, or FSM, CRF, or other quantitative inference network, generally a state machine): starting from an initial / pre-state of the scene where no humans are present and the object inventory and layout are known, we constantly scan successive frames for human presence. The appearance of a human (one or more) - either from a physical entry point at a predefined location (e.g., a door) or from a geometric entry point into a predefined frame (e.g., mid-level on the left or right) - triggers a chronological change of state (CCS). From then on, each person is tracked individually in its own state machine (interactions are possible, see below) until a reverse CCS occurs (i.e. the person leaves the frame). The effect of CCS is always applied with each new person entering and leaving, up to the maximum number of people tracked simultaneously. People may appear accompanied by one or more predefined objects (e.g. briefcase, office cart, file binder, cleaning cart, bat, handgun, etc.). Tracking-by-detection of people is performed standardly between frames. A motion state change (KCS) occurs when a person reaches a new state: a) motion velocity (stationary, walking, running); b) posture (standing, sitting, bending, lying down, shaking) as determined by pose estimation. A proximity state change (PCS) occurs when a person or an untethered object (e.g. desk, computer keyboard, phone, window, doorknob, etc.) reaches a certain distance. A Contact-Constant Change (HCS) occurs due to contact between a registered human and an object, and from now on the two will be considered related.The two remain in that state until separated by the opposing HCS (eg, checking in the briefcase or hanging up the phone).

[0032]

[0520] In a more specific basic method overview based on activity events introduced in 510, the detection of CCS and KCS starts with a contour view showing the contours of both stationary and moving objects (the latter possibly having a speed indicator as defined in module 300). In one embodiment, the contours of moving objects are piped through a standard Histogram-of-Gradients (HOG) calculation and an optimal classification such as a Support Vector Machine (SVM), although other implementations are possible as part of the invention. The result is combined with static contours as needed. In one embodiment, tracking is based on a recursive Kalman filter approach of a type well known to those skilled in the art, simplified by close temporal frame proximity (high frame rate) and multi-scale representation. PCS detection is based on combining the results of the moving and stationary contours. For HCS detection, a basic human limb detection and tracking may be further employed, based on the same principles used for the overall human contour.

[0033]

[0530] In one embodiment of the present invention, EOI can be programmed to be triggered at any node of the state machine 510 under defined conditions (e.g., fall, run, tail, leave briefcase / backpack, call, sit, wave, human group size exceeds limit). Probabilities (or equivalent measures of relevance / likelihood) in the proposed model need to be estimated / measured in advance (e.g., prior probabilities, transition probabilities, conditional probabilities in the described HMM-based embodiment). All such values ​​can be reasonably estimated in ways obvious to those skilled in the art, both from publicly available datasets and from other methods of (un)supervised learning of relevant statistics from new original footage.

[0034]

[0550] An AAA server with standard authentication, authorization, and accounting capabilities: These are commonly known in the art and (in one embodiment) are provided by the FreeRadius software package.

[0035]

[0600] A client portable device or mobile viewer (possibly preceded in the chain by a server entity 550 with specific user account / credentials / distribution capabilities typical of a server) can transmit the outline video (and possible the full video after EOI detection) in real time. This unit may comprise any suitable dashboard-based user interface (UI), but as a minimum should comprise an instance of the decoder 900 to enable said viewing of the outline video and to facilitate real time viewing of the full video transmitted in compressed form following EOI.

[0036]

[0700] An independent network server module that performs scene / object classification and indexing based on the results from module 500. As an independent organizational module, it performs the following functions: facilitating secure uploading, management and archiving of user data files (i.e., data / information provided by module 500), classifying and indexing events / scenes / objects and enabling effective searches based thereon, according to several methods known in the art.

[0037]

[0800] A learning module trained to recognize (or operate on) the units, events, and trends thereof defined and indexed in 500 and 700 has the following primary goals: a) improve recognition of EOI (while monitoring for possible misrecognition, etc.), b) determine longitudinal changes in the observed subjects' Activities of Daily Living (ADL) characteristics such as gait, hip bending, walking, etc., and c) collect non-personally identifiable information for a general database tracking the occurrence of ADLs and EDLs in the monitored population. Similar to the provisions made for component 500 - the primary inputs of such a module in the preferred embodiment tracked here are various data abstracted from video inputs, although its functionality is not limited to that domain and it should be straightforward for one skilled in the art to generalize to include other data inputs (e.g., voice, miscellaneous binary, discrete and continuous valued sensors).

[0038] In another embodiment of

[0800] , a learning module providing "big data analytics", i.e., learning to recognize (or manipulate) the units, events, and their trends (changes over time) defined and indexed in 500 and 700, has the following primary goals: a) to improve recognition of EOI (while monitoring for possible misrecognitions, etc.), b) to determine longitudinal changes in observed subjects' gait, flexion, gait, rise time, and similar Activities of Daily Living (ADL) characteristics, and c) to collect non-personally identifiable information for a general database tracking the occurrence of ADLs and EDLs in the monitored population. Similar to the provisions made for component 500 - the primary input of such a module in the preferred embodiment tracked here is various data abstracted from video inputs, although its functionality is not limited to that domain and it should be straightforward for one skilled in the art to generalize to include other data inputs (e.g., voice, miscellaneous binary, discrete and continuous value sensors).

[0039]

[0850] A web server (network server) which delivers content, if necessary and for a limited time, to the (multiple) mobile viewers 600 in response to requests submitted by such units via standard HTTP protocol.

[0040]

[0900] The decoding / decompression of the video stored in the ring buffer 400 is performed here. This unit can be physically separate and the time when such decompression is performed is independent of the real-time workflow of the input signal described above. In one embodiment, it can be attached to or operate as part of the mobile viewer 600 for the purpose of accessing the full video (FV) image in real time (such as via the BS 500 following an EOI) or following the ring buffer 400 with stored image directly preceding such an EOI. The present invention is not limited to a completely specific codec (compression-decompression) method, but may encompass all such methods from the class of DWT-based subband approaches, including both lossless and lossy variants thereof (most notably the JPEG2000 standard among them): see Bozinovic et al., "A fast DWT-based intermediate video codec optimized for massively parallel architectures" (U.S. Pat. No. 9,451,291) for an implementation example of a dyadic n-level DWT decomposition of the input, as well as other methods well known in the literature.

[0041] The surrounding environment of the human subject includes, but is not limited to, turning lights on and off, opening and closing doors and windows, and changing or moving room furnishings (curtains, chairs, furniture, etc.).

[0042] Sensors used in the system include, but are not limited to, radar, ambient light detectors or level detectors, pressure, temperature, humidity, carbon dioxide, carbon monoxide, and VOC (volatile organic compounds) sensors.

[0043] Anatomy of a typical computing device The present invention refers to a computing program, application or software, all of which are synonymous and used interchangeably. The present invention is applicable to any computing device that is connected to a communications network or the Internet, either wired or wirelessly. An embodiment of the present invention may be implemented by a processor-based computer system. The system includes a database for receiving and storing information from a user, and application software for the user and displaying the feedback information.

[0044] A computer system operates to perform the functions of the server component. The computer system includes a processor, memory and disk storage. The memory stores computer program instructions and data. The processor executes the program instructions or software to process the data stored in the memory. The disk storage stores data that is transferred to and from the memory. It is noted that the disk storage can be used to store data that would typically be stored in a database.

[0045] All of these elements are interconnected by one or more buses, allowing data to be communicated between the elements. Note that the memory is accessible to the processor via the buses and includes an operating system, a program partition, and a data partition. The program partition stores and allows the processor to execute program instructions that implement the respective system functions described herein. The data partition is accessible to the processor and stores data used during execution of the program instructions.

[0046] Memory, flash, disk are machine-readable media and may include any medium capable of storing instructions adapted to be executed by a processor. Examples of such media include, but are not limited to, read-only memory (ROM), random access memory (RAM), programmable ROM, erasable programmable ROM, electronically erasable programmable ROM, dynamic RAM, flash memory, magnetic disks (such as disks and hard drives), optical disks (such as CD-ROMs), optical fibers, electrical signals, lightwave signals, radio frequency (RF) signals, and any other device or signal capable of storing digital information. In one embodiment, the instructions are stored on the medium in compressed and / or encrypted form. As used herein, the phrase "adapted to be executed by a processor" is meant to encompass instructions stored in compressed and / or encrypted form, as well as instructions that must be compiled or installed by an installer before being executed by the processor. Additionally, the system may include various combinations of machine-readable storage devices accessible by the processor and capable of storing a combination of computer program instructions and data.

[0047] The computer system also includes a network interface. The network interface may be any suitable means for controlling communication signals between network devices using a desired set of communication protocols, services, and operating procedures. The communication protocols are arranged in a layered structure, also called a protocol stack, such as the operating system, the CBE communication layer, and the TCP / IP (Transport Control Protocol / Internet Protocol) layer. The network interface may include a connector for connecting the interface with a suitable communication medium. Those skilled in the art will appreciate that the network interface may receive communication signals over any suitable medium, such as twisted pair wire, coaxial cable, optical fiber, radio frequency, etc.

[0048] A typical computer system includes a processor, memory, disk or flash storage, a network interface, and a protocol stack with a CBE communication layer and a TCP / IP layer. These elements operate in a similar manner to their counterparts in a computer system.

[0049] Detailed embodiments of the present invention are disclosed. However, the disclosed embodiments are merely exemplary of the present invention, which may be embodied in various forms. The specific structural and functional details disclosed are not to be construed as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to variously employ the present invention in substantially any suitable detailed structure. The titles, headings, terms, and phrases used are not intended to limit the subject matter or scope, but rather to provide an easily understandable description of the present invention. The present invention is composed of several subparts that independently perform some of the overall functionality of the present invention and contribute to system level functionality when combined with other parts of the present invention. The terms "a" or "an" are defined as one or more. The term "multiple" is defined as two or more than two. The term "another" is defined as: at least a second or more. The terms "including" and / or "having" are defined as "comprising" (i.e., open language). The term "coupled" is defined as connected, although not necessarily directly and not necessarily mechanically connected.

[0050] Any element in a claim that is not expressly recited as a "means for" to perform a particular function or a "step for" to perform a particular function is not to be construed as a "means" or "step" clause under 35 U.S.C. Sec. 112, Paragraph 6. The use of "step of" in a claim is not intended to invoke the provisions of 35 U.S.C. Sec. 112, Paragraph 6.

[0051] Incorporation by Reference All publications, patents and applications, and internet website addresses mentioned are incorporated by reference to the same extent as if each individual document was specifically and individually indicated to be incorporated: U.S. Patent No. 9,916,538 (Zadeh); U.S. Patent No. 10,361,802 (Hoffberg); U.S. Patent Application Publication No. 2014 / 0201126 (Zadeh); U.S. Patent Application Publication No. 2018 / 0204111 (Zadeh); U.S. Patent Application Publication No. 2019 / 0087965 (Datta); U.S. Patent No. 8,814,811, U.S. Patent Application Publication No. 2014 / 0247335 A1; 1.CHAAROUI, et al., “Continuous Human Action Recognition in Ambient Assisted Living Scenarios Lecture Notes of the Institute for Computer Sciences” Social Informatics and Telecommunications Engineering, 2015, 2. Planinc et al., “Computer Vision for Active and Assisted Living” 2016, 3. WANG, et al., "Grading Image Retrieval Based on DCT and DWT Compressed Domains Using Low-Level Features," J. of Communications, Vol. 10, No. 1, January 2015. 4. DELAC, et al., “Towards Face Recognition in JPEG2000 Compressed Domain” 2007, IWSSIP & EC-SIPMCS, Slovenia. 5. MANDAL, et al., "A Critical Evaluation of Image and Video Indexing Techniques in the Compressed Domain," Image and Vision Computing Journal, Vol. 17, No. 7, pp. 513-529, May 1999. 6. SZU, et al., "Wavelet Transforms and Neural Networks for Compression and Recognition," Neural Networks, Vol. 9, No. 4, pp. 695-708, 1996. 7. DEBARD, et al., "Camera-based fall detection using a particle filter," August 25-29, 2015, 2015 Conference Paper, 201537th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy. 8. KHAN, et al., "Video Analytic for Fall Detection from Shape Features and Motion Gradients," Proceedings of the World Congress on Engineering and Computer Science 2009, Volume II, WCECS 2009, October 20-22, 2009, San Francisco, CA, USA. 9.OHM, et al., "Interframe Wavelet Coding - motion picture representation for universal scalability", Signal Processing: Image Communication 19 (2004), pp. 877-908, 10. AGGARWAL, et al., "Human Activity Analysis: A Review", ACM Computing Survey, Vol. 43, No. 3, Article 16, April 2011.

Claims

1. a) using a camera in a computing device with memory storage and power source to capture and store video data; b) simultaneously compressing and analyzing the video data using a low level analysis of contours of stationary and moving objects within the video data; the compression is wavelet-based; The analysis is contour extraction and edge detection; The compression integrates the analysis into the compression workflow. A method for processing video data.

2. The camera is wirelessly connected to a base station and communicates over a two-way full duplex connection. The method of claim 1.

3. The camera outputs and streams outline video data and a tri-level gray image. The method of claim 1.

4. The camera generates and streams three gray level images: a flat background, a stationary object contour, and a moving object contour. The method of claim 1.

5. a learning module that learns to recognize units and events and trends of said units and events over time; The method of claim 1.

6. the learning module learns to recognize long-term changes in activities of daily living, gait, bending, walking, and rise time; The method according to claim 5.

7. the learning module learns to recognize long-term changes in audio data and binary, discrete, and continuous sensor data; The method according to claim 5.

8. the learning module learns to recognize long-term changes in the surrounding environment, lighting, doors, windows, and furniture in the room; The method according to claim 5.

9. the analysis of the video data includes comparing real-time semantic human activities within the video data, whereby the method identifies specific human activities within the video data. The method of claim 1.

10. the video data being a video sequence of a two-dimensional N×K array of values; using a low level of analysis to generate salient contour features of objects and people in moving and stationary scenes, thereby the method provides an automatic scene analysis of the video sequence; The method of claim 1.

11. The method uses real-time compression and automated visual analysis of a source video signal into a two-dimensional array of numbers; The method of claim 10.

Images