Video data processing

Abstract

Video data is processed within a building, e.g. for building control purposes. An object is detected within video data from camera 1901. The camera 1901 may perform the detection. Then video data proc

Description

TECHNICAL FIELD

[0001] The present invention relates to video data processing. Video data processing according to embodiments of the present invention is implemented through a distributed computing architecture or platform that may further implement an Artificial Intelligence (Al) system for performing the monitoring or control of a system within a building. Certain embodiments of the present invention are particularly directed to video data processing within domestic premises such as houses and apartments-that is, homes. BACKGROUND

[0002] Buildings, including homes, offices and commercial or industrial premises, are becoming increasingly complex. A single building, for instance a single home, may include multiple separate building systems each linking together a range of devices. The devices include sensors providing sensor data indicative of a sensed parameter for the building or the building system and components forming part of the building system and providing component data indicative of a component state. As an example, a typical home Heating, Ventilation and Air Conditioning (HVAC) system may include an array of environment sensors, including for instance temperature and humidity sensors distributed through the home. A HVAC system may include an array of controllable components, for instance fans and valves for controlling ventilation, heating, and cooling sub-systems within the home. Clearly, there is a need for some form of control system to control the HVAC system components based on their component state and based on sensing data.

[0003] One particular type of HVAC system (or a component or sub-system within a HVAC system) is a Mechanical Ventilation with Heat Recovery (MVHR) system. MVHR is a type of mechanical ventilation system used in residential and commercial buildings to provide fresh air while recovering the heat from the outgoing air. Extracted stale air from the building passes through a first duct network to a heat exchanger which transfers the heat from the outgoing air to the incoming fresh air, which is then distributed back into the building through a separate duct network. This process helps to maintain a comfortable indoor temperature while reducing energy consumption and carbon emissions. The MVHR system also helps to improve indoor air quality by continuously exchanging stale air with fresh air, reducing the build-up of pollutants and excess moisture. It will be appreciated that an MVHR system requires a control system for its respective parts (which may include at a minimum pumps or fans and optionally control valves. A typical home may include numerous further building systems (or sub systems), for instance security systems, health care or health monitoring systems and energy supply and control systems, each with their own specific requirements for sensors, controllable components, and control mechanisms (though in some cases there may be overlap in terms of shared sensors).

[0004] Commercially deployed building control systems are often limited in terms of their performance. They may be relatively “dumb” in the sense that they are purely reactive. For instance, if the sensed temperature is below a set threshold, then a heating system may be turned up. Typically, there is no capability for building systems to learn and adapt to the specific attributes of a building, nor to adapt to the requirements of the building occupants.

[0005] There is an industry trend towards “smarter” buildings. Smarter in the sense of better able to react and adapt to the attributes of a building and to adapt to the needs of the building occupants. Al holds much promise for developing smarter buildings. Particularly the use of machine learning or deep learning to analyse real life data for a building and its building systems, to train suitable machine learning algorithms for the building systems and to apply those trained algorithms to provide for control of building systems that are better adapted to meet the needs of the building occupants. Al may improve building systems with the result that the buildings themselves are more energy efficient, secure, comfortable, or healthier for their occupants.

[0006] While the majority of modern buildings have an internet gateway provided by an Internet Service Provider (ISP), those gateways typically have no inbuilt Al inference acceleration capability. Neither is it commonplace for Al hardware capability to be separately provided within buildings (at least for homes - in commercial and industrial settings the use of Al to control building systems is more well established). With the rise of web 3.0 (also known as the “Semantic Web”) it is expected that the need for Al capability within the home will increase. Web 3.0 refers to the next evolution of the World Wide Web, in which data is interconnected and machine-readable, allowing for more intelligent and efficient use of information. Web 3.0 is also expected to provide greater privacy and security for users, as well as more personalized and relevant experiences for users.

[0007] In the interim, before widespread deployment of Al hardware capability to homes, any Al task or insight, including for the control of building systems, is normally carried out using cloud computing. Cloud computing refers to the delivery of computing services such as servers, storage, databases, software, and networking over the internet. These services are provided by remote servers that are maintained by a cloud service provider, allowing users to access and use them on-demand without needing to own and manage their own infrastructure. Cloud computing offers advantages such as cloud scalability, flexibility, and cost-efficiency. While cloud computing offers numerous benefits, there are some potential problems and challenges to consider, including: • Security: Storing data and applications in the cloud can potentially expose them to security risks, such as data breaches or unauthorized access, if the appropriate security measures are not implemented. For a home control system this data could include personally identifiable or sensitive information such as whether the building is occupied. • Availability: Cloud services depend on reliable internet connectivity, so disruptions to the internet or to the cloud service provider’s infrastructure can result in downtime. For instance, for control of a home HVAC system such interruption may be unacceptable. • Vendor lock-in: Switching cloud providers can be difficult and expensive, so customers may feel “locked in” to their current provider. • Cost: While cloud computing can be cost-efficient, the costs can add up over time, especially if users are not careful to monitor and manage their usage. • Latency and bandwidth can potentially be a problem for cloud computing, depending on the specific application or use case.

[0008] As a result of these factors, particularly latency and bandwidth, cloud computing may not provide an acceptable Quality of Service (QoS) for Al based control of building systems.

[0009] It is an aim of certain examples of the present invention to solve, mitigate or obviate, at least partly, at least one of the problems and / or disadvantages associated with the prior art. Certain examples aim to provide at least one of the advantages described below. BRIEF SUMMARY OF THE INVENTION

[0010] According to a first aspect of the present invention there is provided a computer implemented method of processing video data within a building, the method comprising: responsive to the detection of an object within video data, performing first video data processing upon the video data at an edge computing device associated with the building, wherein the first video data processing comprises further classification of the detected object or classification of a further object within the video data; determining, at the edge computing device, based on the first video data processing, whether second video data processing of the video data is required; and if it is determined that second video data processing is required, performing second video data processing at a fog computing device associated with the building; wherein the second video data processing determines whether to generate an output signal or a control signal for a building system based on the video data, the detected object, and the results of the first video data processing.

[0011] The first video data processing may further comprise image segmentation. The second video data processing may further comprise context aware or situation aware processing of the video data.

[0012] The method may further comprise receiving, at the edge computing device, the video data, and the indication of a detected object within the video data from a video camera associated with the building.

[0013] The method may further comprise receiving, at the edge computing device, the video data from a video camera associated with the building; and processing the received video data at the edge computing device to detect an object within the video data.

[0014] An object detected within the video data may comprise one or more of: a person; a vehicle; or an animal.

[0015] If the detected object comprises a person, the first video data processing may comprise further classification of the detected person as one of: a child; a teenager or young person; an adult; or an elderly person.

[0016] If the detected object comprises a person, the first video data processing may comprise detecting and classifying one or more further object within the video data; and wherein the one or more further object comprises one or more of: a food item held by the detected person; or a potential weapon held by the detected person.

[0017] If the detected object comprises a person, the first video data processing may comprise identifying one or more body part visible in the video data from a group including: an arm; a leg; or a hand.

[0018] The method may further comprise training a first machine learning algorithm using video data and indications of detected objects within the video data at the fog computing device; and providing the trained first machine learning algorithm to the edge computing device; wherein the first video data processing comprises processing the video data using the trained first machine learning algorithm.

[0019] According to a second aspect of the present invention there is provided a computer-readable storage medium having computer-readable program code stored therein that, in response to execution by a processor forming part of an edge computing device and a processor forming part of a fog computing device, cause the edge computing device and the fog computing device to perform the above described methods.

[0020] According to a third aspect of the present invention there is provided a system for processing video data within a building, the system comprising: an edge computing device; and a fog computing device; wherein the edge computing device is configured to: responsive to the detection of an object within video data, perform first video data processing upon the video data, wherein the first video data processing comprises further classification of the detected object or classification of a further object within the video data; and determine, at the edge computing device, based on the first video data processing, whether second video data processing of the video data is required; and wherein the fog computing device is configured, if it is determined that second video data processing is required, to perform second video data processing; wherein the second video data processing determines whether to generate an output signal or a control signal for a building system based on the video data, the detected object, and the results of the first video data processing.

[0021] According to a fourth aspect of the present invention there is provided a computer implemented method of operating an edge computing device to process video data within a building, the method comprising: responsive to the detection of an object within video data, performing first video data processing upon the video data, wherein the first video data processing comprises further classification of the detected object or classification of a further object within the video data; and determining, at the edge computing device, based on the first video data processing, whether second video data processing of the video data is required.

[0022] According to a fifth aspect of the present invention there is provided a computer-readable storage medium having computer-readable program code stored therein that, in response to execution by a processor forming part of an edge computing device, cause the edge computing device to perform the above described method.

[0023] According to a sixth aspect of the present invention there is provided an edge computing device comprising a processor and a memory storing executable instructions that, in response to execution by the processor, cause the edge computing device to perform the above described method.

[0024] According to a seventh aspect of the present invention there is provided a computer implemented method of operating a fog computing device to process video data within a building, the method comprising: responsive to a determination, at an edge computing device, based on first video data processing, that second video data processing of the video data is required, performing second video data processing at a fog computing device associated with the building; wherein the second video data processing determines whether to generate an output signal or a control signal for a building system based on the video data, the detected object, and the results of the first video data processing.

[0025] According to an eighth aspect of the present invention there is provided a computer-readable storage medium having computer-readable program code stored therein that, in response to execution by a processor forming part of a fog computing device, cause the fog computing device to perform the above described method.

[0026] According to a nineth aspect of the present invention there is provided a fog computing device comprising a processor and a memory storing executable instructions that, in response to execution by the processor, cause the fog computing device to perform the above described method.

[0027] There is further disclosed a computer implemented method of monitoring or controlling a system within a building, the method comprising: training a machine learning algorithm using device data from at least one device relating to the building or forming part of a building system at a fog computing device associated with the building; receiving device data from the at least one device at an edge computing device associated with the building; and processing the received device data at the edge computing device using the trained machine learning algorithm to generate an output signal relating to the building system or a control signal for the building system.

[0028] Advantageously, control or monitoring of building systems (particularly for a smart home) may be improved using an Al distributed architecture in which training of a machine learning algorithm is performed by a fog computing device and execution of the trained algorithm is performed by an edge computing device. The principle advantage of this is that the Al distributed architecture is scalable and extensible: the more complex / expensive fog computing device can perform training for multiple machine learning algorithms associated with different building systems, while the trained machine learning algorithm for each building system (for instance, air quality, security or health monitoring) can be executed on a respective edge device which communicates with an appropriate set of devices such as sensors and controllable system components to receive data, make predictions and / or control devices.

[0029] This split between fog computing and edge computing gives benefits in terms of cost reduction (where there are multiple systems in a building needing separate control), extensibility / scalability through the ability to add additional edge computing devices to expand to control additional building systems, while leveraging the investment in a single (or reduced number) of fog computing devices to provide Al algorithm training, future proofing by permitting as yet unanticipated future building systems to be controlled through the same Al distributed architecture, cost effectiveness, security, privacy and the ability to additionally communicate with cloud computing services due to the edge / fog implementation (for instance, for data collection, fault reporting, communication with other external systems).

[0030] The Al distributed architecture allows for more efficient control of building systems compared to conventional alternatives. As an example, for a HVAC building system (or particularly an MVHR system) may be optimised to the predicted occupancy level of the building thereby reducing excess heating (for instance when it is predicted that the building will be empty) which can reduce the carbon intensity of the building. A further example is that when it is predicted that the building will be occupied at a future time point then the building may be preferentially heated during time periods when the electricity supply is anticipated to be predominantly supplied by renewable or low carbon energy sources, or when there is typically overcapacity on the electricity supply network (which may also correspond to periods of reduced cost electricity supply).

[0031] The at least one device may comprise at least one of: a sensor providing sensor data indicative of a sensed parameter for the building or the building system; or a component forming part of the building system and providing component data indicative of a component state. There is no limitation to any particular type of device: each device may be anything which can provide device data indicative of a sensed parameter within a building or within or part of a building system, ora device forming part of a building system able to provide data indicative of a device state. Advantageously, existing, or newly deployed devices may be used within a building to obtain data indicative of how a building or system functions or operates.

[0032] The output signal may comprise at least one of: an output message or alert to a user indicating a state of the building or the building system or if the state or the building or the building system is outside of a defined range; or a prediction of a current or future state of the building or the building system. There is no limitation to any particular type of output, nor any particular delivery mechanism for an output signal.

[0033] The control signal may comprise a signal instructing a component setting or change of operation for a component forming part of the building system either immediately or a future point in time. Advantageously, not only can building systems be monitored, but they can also be directly controlled (for instance in response to a predicted building or system state) so as to permit the building system to run more effectively. Effective building control can include energy efficiency, improving the comfort and health of building occupants and control that prolongs the lifespan of a building system by avoiding operating the system when it is unnecessary.

[0034] The method may further comprise: the edge computing device storing device data received from the at least one device. Where device data is stored this may be for the purpose of its later use when executing a trained Al model (such that the model may process both live device data and historic device data).

[0035] The method may further comprise: the edge computing device forwarding device data received from the at least one device to the fog computing device fortraining the machine learning algorithm or updating the training of the machine learning algorithm. There is no limitation to any specific routing or topology for distributing data around the distributed Al architecture.

[0036] The method may further comprise: the edge computing device storing device data received from the at least one device and periodically forwarding stored device data to the fog computing device.

[0037] The method may further comprise: the fog computing device training the machine learning algorithm using a training set of device data received from the at least one device, the training set of device data being received directly from the at least one device or via the edge computing device. The volume of device data required to form an effective training set may be context dependent for the building system being observed and the building within which it is deployed.

[0038] The method may further comprise: updating the training of the machine learning algorithm at the fog computing device using device data received from the at least one device; and providing an updated trained machine learning algorithm to the edge computing device for use in processing further received device data. Advantageously this allows the fog computing device to learn how effectively building systems are controlled or monitored and build this feedback, into the training of an updated model.

[0039] Updating of the training of the machine learning algorithm may further use the training set of device data or previously received device data.

[0040] Updating of the training of the machine learning algorithm may be performed continuously as device data is received from the at least one device, the updated trained machine learning algorithm being provided to the edge computing device either continuously or periodically; or wherein updating of the training of the machine learning algorithm is performed intermittently or periodically using device data received from the at least one device.

[0041] The trained machine learning algorithm may comprise at least one of: a predictive machine learning model adapted to provide a prediction of a current or future state for the building or the building system; or an inference machine learning model adapted to provide a control signal instructing a component setting or a change of operation for a component forming part of the building system either immediately or a future point in time to control or adjust a predicted state for the building system. The trained machine learning algorithm may further comprise an observational or reinforcement learning model that takes account of feedback, concerning operation of the models for controlling or monitoring building systems.

[0042] Alternatively, both a predictive machine learning model and an inference machine learning model may be implemented at the edge computing device; the predictive machine learning model providing a prediction of a current or future state for the building system and the inference machine learning model provides an output to a component forming part of the building system to control or adjust the building system according to the prediction from the predictive machine learning model.

[0043] The at least one device may comprise a sensor providing sensor data indicative of a sensed parameter for the building or the building system; wherein at least one component forming part of the building system provides component data indicative of a component state to the fog computing device or to the edge computing device; and wherein the machine learning algorithm is trained at the fog computing device using sensor data and component data.

[0044] The machine learning algorithm may comprise an inference machine learning model adapted to provide a control signal instructing a component setting or a change of operation for the at least one component forming part of the building system either immediately or a future point in time to control or adjust a predicted state for the building system.

[0045] The method may further comprise the fog computing device storing sensor data received from the at least one sensor or component data from the at least one component.

[0046] Both a predictive machine learning model and an inference machine learning model may be implemented at the edge computing device; and the predictive machine learning model may provide a prediction of a current or future state for the building system and the inference machine learning model may provide an output to the at least one component forming part of the building system to control or adjust the building system according to the prediction from the predictive machine learning model.

[0047] The edge computing device may comprise a sensing edge computing device receiving sensor data from the at least one sensor within or associated with the building; wherein the method may further comprise a system edge computing device communicating with the at least one component to receive component data; wherein the sensing edge computing device processes the received sensor data using a predictive machine learning algorithm, trained by the fog computing device, to provide a prediction of a current or future state for the building or the building system and to provide an output to the system edge computing device if the prediction is outside of a defined range; and wherein the system edge computing device processes received component data and the output received from the sensing edge computing device using an inference machine learning algorithm, trained by the fog computing device, to provide a control signal instructing a component setting or a change of operation either immediately or a future point in time to control or adjust a predicted state for the building system.

[0048] The method may further comprise: the system edge computing device observing building system state data or control data, and providing this data to the fog computing device; the fog computing device training an observational or reinforcement learning model and providing the trained model to the system edge; and the system edge executing the trained observational or reinforcement learning model to adjust at least one threshold or range for a prediction output for the predictive model.

[0049] Advantageously, the observational or reinforcement learning model observes and records the states and actions of components of the building system, for instance a ventilation system, and the states and actions of the building occupants. The observational or reinforcement model determines what patterns exist within the data, but is not responsible for controlling the building system components devices itself to effect system change. This permits context awareness or situational awareness by adjusting the parameters of the predictive model based on real-time observations of the control of the building system which is fed back, to the operation of the predictive model. It may be assumed that if building occupants do not override any such changes through a feedback mechanism then implicitly those changes are approved.

[0050] The method may further comprise: the system edge computing device observing building system state data or control data, and providing this data to the fog computing device; the fog computing device training an observational or reinforcement learning model and providing the trained model to the system edge; and the system edge executing the trained observational or reinforcement learning model to adjust at least one threshold or range fora prediction output for the predictive model.

[0051] That is, in some embodiments the functions of Al prediction and Al inference may be separated out. The sensing edge may be responsible for monitoring the building system and forming prediction of future states for the building or the building system. The system edge may be responsible for building system control: adjusting control data sent to components within the building system, optionally based on feedback on predictions from the sensing edge.

[0052] For an example application of monitoring air quality and controlling a ventilation system to ensure that air quality stays within defined limits, the sensing edge computing device may comprise an environment sensing edge computing device receiving environment sensor data from at least one environment sensor within or associated with the building. The system edge computing device may comprise a ventilation system edge computing device communicating with at least one component forming part of a building ventilation system to receive component data indicative of a ventilation system component state; wherein the environment sensing edge computing device processes the received environment sensor data using a predictive machine learning algorithm, trained by the fog computing device, to provide a current or a future air quality prediction for the building and to provide an output to the system edge computing device if the prediction is outside of a defined range; and wherein the system edge computing device processes received ventilation system state data and the output received from the environment sensing edge computing device using an inference machine learning algorithm, trained by the fog computing device, to provide a control signal instructing a ventilation system component setting or change of operation either immediately or a future point in time to control or adjust a predicted state for the ventilation system.

[0053] The at least one environment sensor may comprise one or more of: a temperature sensor; a humidity sensor; a particulate matter sensor; a gas sensor; or a pressure sensor. This should not be considered to be an exhaustive list of available environment sensors that may yield data relevant to air quality.

[0054] The environment sensing edge computing device may further configured to receive sensor data from one or more of: an occupancy sensor; or a camera. The skilled person will be aware of other ways in which occupancy may be monitored and other sources of external data that may prove relevant to monitoring and controlling air quality.

[0055] The at least one ventilation system component may comprise one or more of: a ventilation fan; a ventilation valve; a heating control or valve; or an air conditioning control or valve. This is not an exhaustive list of components of a HVAC system that may be controlled and may provide data indicative of a system state.

[0056] The edge computing device may comprise a first edge computing device processing received device data using a first trained machine learning algorithm to generate an output signal or a control signal for a first building system; wherein the method further comprises at least one further edge computing device processing received device data from at least one device relating to the building or forming part of a further building system using a further trained machine learning algorithm to generate an output signal, relating to the building or the further building system, or a control signal for the further building system; and wherein both the first machine learning algorithm and the further machine learning algorithm are trained using device data at the fog computing device. Accordingly, the benefits of scalability and extensibility provided by the distributed Al architecture may be provided by deploying further edge computing devices for monitoring or controlling further building systems.

[0057] Each of the first and further building systems may comprise separate building systems selected from a group including: a building ventilation system; a building heating and ventilation system; a building heating, ventilation, and air conditioning system; a building security system; a building energy use monitoring and control system; or a healthcare system.

[0058] For at least one further building system a machine learning algorithm trained by the fog computing device may be executed on the fog computing device for processing device data received by the fog computing device. That is, while the primary function of the distributed Al architecture is to separate the roles of model training and model executing, in some cases particularly for complex multi-system deployments it may be more efficient for at least one model to be executed as well as trained by the fog computing device.

[0059] The method may further comprise at least one edge computing device or fog computing device: receiving device data from at least one controllable device within or associated with the building; processing the device data using a control algorithm; and returning a control signal to the controllable device. That is, while the distributed Al architecture serves to control or monitor complex building systems, the available computing power may also be leveraged to control simpler devices or appliances within a building.

[0060] The at least one controllable device may comprise at least one of: a kitchen appliance, for instance an oven, microwave, refrigerator, freezer, or dishwasher; ora utility appliance, for instance a clothes washing machine or tumble drier; a home entertainment device, for instance a television or other audio or visual device. This is not an exhaustive list.

[0061] At least one edge computing device or fog computing device may receive data from a system external to the building for processing using a trained machine learning algorithm together with device data, or for use in training a machine learning algorithm at the fog computing device.

[0062] The data from the system external to the building may comprise at least one of: current or predicted meteorological information; and current or predicted location information for building occupants. Other pertinent external data sources will be readily apparent to the skilled person from consideration of a particular building system to be monitored or controlled.

[0063] Current or predicted location information for building occupants may be derived from at least one of: calendar information; and mobile device location services.

[0064] The output signal may be provided through at least one of: a local area network portal; a mobile device application; ora display or touchscreen device within the building.

[0065] Each edge computing device, the fog computing device, and any devices, sensors or system components may be communicatively coupled either through a wired network within the building or wirelessly.

[0066] Communications between each edge computing device, the fog computing device, and any devices, sensors or system components may be compliant with the Matter protocol. This is only one example of a suitable communication protocol, albeit one that permits the use of Matter compatible commercially available sensors and system components.

[0067] Wireless communications may be transmitted through a Thread network, particularly where Matter compatible devices are to be used.

[0068] The building may comprise a domestic premise. While the present invention is broadly applicable to any building, as discussed below in connection with figures 7a and 7b the present invention is particularly suitable for monitoring or controlling building systems within a home.

[0069] There is further disclosed a computer-readable storage medium having computer-readable program code stored therein that, in response to execution by a processor forming part of an edge computing device and a processor forming part of a fog computing device, cause the edge computing device and the fog computing device to perform the method of any one of the preceding claims.

[0070] There is further disclosed a system for monitoring or controlling a system within a building, the system comprising: at least one device relating to the building or forming part of a building system and configured to provide device data; an edge computing device configured to receive device data from the at least one device and process the received device data using a trained machine learning algorithm to generate an output signal relating to the building system ora control signal for the building system; and a fog computing device configured to train the machine learning algorithm.

[0071] The system described above may be further configured to execute any part of the above-described methods of monitoring or controlling a building.

[0072] There is further disclosed a computer implemented method of operating an edge computing device to monitor or control a system within a building, the method comprising: receiving device data from at least one device relating to the building or forming part of a building system at the edge computing device, the edge computing device being associated with the building; and processing the received device data at the edge computing device using a trained machine learning algorithm to generate an output signal relating to the building system or a control signal for the building system; wherein the machine learning algorithm is trained using device data at a fog computing device associated with the building.

[0073] The edge computing device may be further configured to perform any part of the above-described methods of monitoring or controlling a building.

[0074] There is further disclosed a computer-readable storage medium having computer-readable program code stored therein that, in response to execution by a processor forming part of an edge computing device, cause the edge computing device to perform the above-described method of operating an edge computing device.

[0075] There is further disclosed an edge computing device comprising a processor and a memory storing executable instructions that, in response to execution by the processor, cause the edge computing device to perform the above-described method of operating an edge computing device.

[0076] There is further disclosed a computer implemented method of training a machine learning algorithm for monitoring or controlling a system within a building, the method comprising: receiving device data from at least one device relating to the building or forming part of a building system; forming a training data set from the device data; and training a machine learning algorithm using the training data set; wherein the trained machine learning algorithm is provided to an edge computing device for processing received device data at the edge computing device to generate an output signal, relating to the building or the building system, ora control signal for the building system.

[0077] Training the machine learning algorithm using device data at the fog computing device may comprise processing a training set of device data using machine learning or deep learning, including one or more steps from a group including: classification; segmentation; correlation; and prediction.

[0078] There is further disclosed a computer-readable storage medium having computer-readable program code stored therein that, in response to execution by a processor forming part of a fog computing device, cause the fog computing device to perform the above-described method of training a machine learning algorithm.

[0079] There is further disclosed a fog computing device comprising a processor and a memory storing executable instructions that, in response to execution by the processor, cause the fog computing device to perform the above-described method of training a machine learning algorithm.

[0080] The term “building” should be broadly construed to mean any structure defining a space which is wholly or partially contained. The term “building” includes industrial, commercial, retail and office buildings or structures as well as domestic premises including houses and apartments (hereinafter referred to as “homes”). However, as will become apparent from the following description, building systems within homes may suitably be monitored or controlled. This is because, as noted above, up until now homes have not typically been provided with Al hardware capability (particularly, Al inference acceleration capability), resulting in home systems being reliant on cloud computing for advanced system control functions. Furthermore, embodiments facilitate context awareness and situational awareness as part of Al control of building systems. As but one example, an embodiment may take account of current and predicted future occupancy information when controlling a building system so that the building system may be optimised to meet the needs of the building occupants (both for total occupancy levels and for the specific requirements of identified occupants). That is, embodiments can observe and predict the behaviour of occupants of a building and take this into account when controlling a building system. For instance, for controlling a building ventilation system, it may be observed that certain occupants cook food or use bathrooms at certain times of day, meaning that it can be predicted that the air quality within the home may be affected at a future point in time and the ventilation system controlled to address this predicted reduction in air quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Examples of the invention are further described hereinafter with reference to the accompanying drawings, in which:

[0082] Figure 1 is a process flow diagram for a distributed Al architecture comprising an edge computing device and a fog computing device for monitoring or controlling a building system;

[0083] Figure 2 is a process flow diagram for an embodiment comprising a modification to the distributed Al architecture of figure 1 in which both a predictive machine learning model and an inference machine learning model are executed upon the edge computing device;

[0084] Figure 3 is a process flow diagram for an embodiment comprising a modification to the distributed Al architecture of figure 1 in which both sensor data and data from a component forming part of a building control system are processed;

[0085] Figure 4 is a process flow diagram for an embodiment comprising a modification to the distributed Al architecture of figure 2 in which both sensor data and data from a component forming part of a building control system are processed;

[0086] Figure 5 is a process flow diagram for an embodiment comprising a modification to the distributed Al architecture of figure 4 in which the distributed Al architecture further comprises both a sensing edge computing device which executes a predictive machine learning model and system edge computing device which executes an inference machine learning model, the edge computing devices processing both sensor data and data from a component forming part of a building control system;

[0087] Figure 6 is a process flow diagram for an embodiment corresponding to the distributed Al architecture of figure 5 for the example of monitoring or controlling a ventilation system;

[0088] Figure 7a is a perspective view and figure 7b is a cross section plan view of a domestic premise, particularly a home (specifically a home constructed as a hybrid building) incorporating a distributed Al architecture according to an embodiment of the present invention;

[0089] Figure 8 is a flow chart illustrating a process of training and implementing a machine learning algorithm for monitoring or controlling a building system where a distributed Al architecture is deployed within a building;

[0090] Figure 9 is a modification to the flow chart of figure 8 further illustrating a method of updating the training of a machine learning algorithm for monitoring or controlling a building system;

[0091] Figure 10 is a modification to the flow chart of figure 8 further illustrating a more detailed method of training of a machine learning algorithm for monitoring or controlling a building system;

[0092] Figure 11 is a flow chart illustrating a method of operating an edge computing device for monitoring or controlling a building system;

[0093] Figure 12 is a flow chart illustrating a method of operating a fog computing device fortraining a machine learning algorithm for monitoring or controlling a building system;

[0094] Figure 13 is a network topology diagram for a distributed Al architecture;

[0095] Figure 14 illustrates an edge computing device forming part of a distributed Al architecture;

[0096] Figure 15 illustrates a fog computing device forming part of a distributed Al architecture;

[0097] Figure 16 illustrates a building or building system sensor device configured to communicate sensor data to a distributed Al architecture;

[0098] Figure 17 illustrates a building system component configured to communicate component data to a distributed Al architecture;

[0099] Figure 18 is a process flow diagram for an embodiment corresponding to a modified form of the distributed Al architecture of figure 6 for the example of monitoring or controlling a ventilation system;

[00100] Figure 19 illustrates a conventional video processing system;

[00101] Figure 20 illustrates a video processing system according to an embodiment of the present invention; and

[00102] Figure 21 is a flow chart illustrating a video processing method according to an embodiment of the preset invention.

[00103] In the following description of the drawings, where corresponding features are shown, the same reference numbers are used. DETAILED DESCRIPTION

[00104] There is disclosed herein a novel distributed Al architecture for monitoring or controlling a building system in which a machine learning model training is performed by a fog computing device and execution of the trained machine learning algorithm is performed by an edge computing device. In particular, but not exclusively, this Al distributed architecture may be deployed within a smart home interacting with a broad range of devices including sensors for sensing parameters of the building such as environmental parameters. The devices may further comprise system components, such as controllable system actuators, capable of providing sensor information or component state information. In some cases, the components may also be configured to receive control signals output from the trained machine learning algorithm for controlling the building system.

[00105] As previously noted, while it is known for building systems to be controlled through cloud computing, it has been identified that particularly latency and bandwidth issues relating to conventional implementations of cloud computing may not provide an acceptable quality of service for monitoring or controlling building systems. Edge computing addresses at least some of the issues associated with cloud computing by bringing data processing closer to the sources of data.

[00106] Edge computing is a distributed computing paradigm that involves processing and analysing data at or near the edge of the network, where the data is generated, rather than relying solely on a centralized cloud computing infrastructure. This enables faster response times, reduced network bandwidth, improved reliability, and enhanced security and privacy for applications and services that require real-time or low-latency processing.

[00107] This may mean processing building system data to provide output signals and control signals for the building system using an edge computing device in or associated with the home. An edge computing device may comprise a relatively low-cost computing device that has the capacity to receive and process device data by executing a trained machine learning algorithm (also referred to herein as a trained Al model or simply trained model or model). The edge computing device may receive data from one or a plurality of devices, process the data and generate an output signal or a control signal. In particular, the edge computing device may incorporate the Al acceleration (particularly, in some embodiments, Al inference acceleration) hardware capability that is necessary to execute a trained Al model. As noted above, conventionally most buildings, especially homes, lack such Al acceleration hardware capability. However, an edge computing device may lack the computational capacity to train a machine learning algorithm in order to generate a trained Al model.

[00108] Fog computing is a distributed computing paradigm that extends the capabilities of edge computing by adding an intermediate layer of computing and storage resources between the edge devices and the cloud. The fog layer is typically located closer to the edge devicesthan the cloud, and provides more powerful computing resources and storage than the edge devices. This allows for more complex processing and analysis of data at the edge. Fog computing is especially useful in applications where data must be processed and analysed in real-time, or where there are limitations on network bandwidth or latency.

[00109] A fog computing device is provided in or associated with the home and communicates with one or more edge computing devices. The fog computing device may be a relatively high-cost computing device that provides the capability fortraining machine learning algorithms for each edge computing device. The fog computing device may also provide orchestration and coordination services for the distributed Al architecture. Each edge computing device may be responsible for processing device data using a trained Al model fora separate building system. Each edge computing device may be responsible for monitoring or controlling a single building system, or potentially a relatively small number of building systems. The devices supplying device data to each edge computing device may be unique to each building system or there may be overlap with a device supplying device data for use in monitoring or controlling two or more building systems (on one or more edge computing devices). This use of a single, powerful fog computing device with a plurality of edge computing devices to monitor or control disparate building systems provides a scalable distributed Al architecture computing platform that may be extended through the provision of additional edge computing devices as required as new building systems are added or expanded.

[00110] Referring now to figure 1, this illustrates a device 101 and a distributed Al architecture comprising an edge computing device 102 and a fog computing device 103. For simplicity of explanation, in figures 1 to 6 only a single edge computing device 102 is illustrated. That is, these figures illustrate simplified deployment scenarios in which only a single building system is to be controlled or monitored using a single edge computing device 102. However, from the discussion above it will be apparent to the skilled person how this is scalable to the fog computing device 103 communicating with and providing machine learning algorithm training for multiple edge computing devices 102 associated with multiple building systems.

[00111] The edge computing device 102 communicates with at least one device 101 in order to receive device data. For instance, a device 101 may comprise a sensor, in which case the edge computing device 102 may be referred to as a sensing edge computing device 102. A sensor 101 may provide sensor data indicative of a sensed parameter for the building or the building system. In figure 1 for simplicity the edge computing device 102 is labelled simple Edge. A sensor 101 may for instance be an environment or air quality sensor 101, in which case the edge computing device may be referred to as an environment sensing edge computing device 102 (discussed in greater detail below in connection with figure 6). As is illustrated in figure 1, there may be only a single device 101 sending device data to the edge computing device 102. However, there may be a plurality of devices 101 providing separate device data to the edge computing device 102, as indicated by the dashed box Device x.

[00112] Figure 1 (and also figures 2 to 6 and 18, described below) are process flow diagrams in which system components (particularly the distributed Al architecture comprising the edge computing device 102 and the fog computing device 103) are illustrated at the top and by descending vertical lines. The exchange of information between system components is illustrated by interconnecting lines. Processes happening at each system component are illustrated upon the descending lines. These process flow diagrams may also be referred to as ladder diagrams. It should not be assumed that the exchange of information and processing illustrated define a strict chronology descending downwards, except where this is required or made apparent in the following description.

[00113] As well or instead, at least one device 101 may be a building system component. For instance, the building system may be ventilation system. It will be understood that in the context of the present document the term ventilation system is intended to encompass building systems including one or more element of heating and air conditioning as well as ventilation. A HVAC system, including an MVHR system, remains a ventilation system. A component of a ventilation system may include a fan arranged to provided component state information such as fan speed or airflow to the edge computing device. In some cases, as will be discussed below, such a system component may be configured to receive a control signal from the edge computing device operating a trained machine learning algorithm, particularly an inference model that has identified the component as an influence for the building system and seeks to control the building system through controlling the component. Where the device 101 comprises a system component the device data may comprise component data indicative of a component state. The component may include a sensor providing the component state information. The component state may indicate a setting or mode of operation for the sensor.

[00114] In some embodiments where a plurality of devices is supplying device data for monitoring or controlling a building, the devices may include both sensors (for instance, supplying data indicating an environment parameter for the building) and system components relating to the building system being controlled or monitored. Examples of this are presented below in connection with figures 3 to 6.

[00115] As already noted, the edge computing device 102 receives device data from the at least one device 101 as indicated by arrows 104 (with the optional supply of device data from further devices indicated by dashed line 104). Only a single transfer of device data 104 is illustrated. However, it will be appreciated that the devices 101 may continuously, regularly, or periodically transfer device data to the edge computing device 102, and this may continue in parallel (that is, simultaneously or concurrently) to the other processes described below. This device data 104 is processed by the edge computing device 102 using a trained machine learning model that executes at step 109 on the edge computing device 102 to provide a model output, for instance an alert or message to a building occupant, or a control signal, for instance to control a component such as a fan. As will be well understood by the skilled person, a machine learning algorithm operates to map an input (or a series of inputs) to an output (for instance, a prediction of a future building state). Particularly, the trained machine learning algorithm executed by the edge computing device to process the received device data may be one or typically both of a predictive machine learning model and an inference machine learning model. Deep learning is a particular subfield of machine learning (and both are subsets of Al) that involves training artificial neural networks, which are composed of layers of interconnected nodes, to learn from vast amounts of data. Deep learning algorithms are typically used for tasks such as image recognition, natural language processing, and speech recognition, where they have achieved state-of-the-art results. Deep learning models are more complex than traditional machine learning models and require large amounts of data and computational power to train effectively. For the control of building systems, the present inventor has identified that deep learning is particularly suited, however in this document the generic term machine learning is used.

[00116] A predictive machine learning model is a type of machine learning model that uses historical data to make predictions or forecasts about future events or trends. These models use algorithms to analyse large amounts of data, identify patterns and correlations, and create a mathematical model that can be used to make predictions based on new data. An inference machine learning model is a machine learning model that has been trained on a dataset to learn to identify patterns and correlations in the data that allow it to identify factors within a system that influence a system state. Either or both of a predictive machine learning model (referred to herein for simplicity as a predictive model) and an inference machine learning model (referred to herein for simplicity as an inference model) may be executed by an edge computing device 102 for processing device data to monitor or control a building system. Figure 2 presents an example in which a single edge computing device implements both predictive and inference machine learning models and figure 5 presents an example in which the respective model types are implemented by separate edge computing devices.

[00117] A predictive machine learning model executed by the edge computing device 102 may process received device data 104 in order to make a prediction of a current or future building system state. An inference machine learning model executed by the edge computing device 102 may process received device data, including building system component state information and identify how building system components should be controlled in order to influence a predicted system state (for instance, a prediction of a current or future system state provided by a predictive machine learning algorithm). In its broadest sense however, the edge computing device 102 may operate a generic trained machine learning algorithm capable of providing a prediction ora control signal for an influencing a system state, and the distinction between predictive and inference machine learning algorithms may be arbitrary or not apparent.

[00118] Prior to execution 109 of a trained model, the fog computing device 103 performs the training of the machine learning algorithm. This training process is based upon a training data set of device data. As illustrated in figure 1, the edge computing device 102 may pass through the device data to the fog computing device 103 at step 106. Optionally (as indicated by the dashed box 105) the edge computing device 102 may also store the device data, particularly in the event that later execution 109 of the trained machine learning algorithm makes use of historic as well as current device data.

[00119] As illustrated, the or each device 101 associated with a particular building system provides device data at step 104 to the edge computing device 102, and this passes the device data through to the fog computing device 103 as required at step 106. This pass through of the device data may occur in real time - that is the edge computing device 102 forwards the device data to the fog computing device 103 substantially as it is received. In other embodiments, the transfer of device data may occur intermittently, for instance once a predetermined buffer of device data is filled. In some embodiments device data may only be provided to the fog computing device 103 in order to form a training data set. Once the machine learning algorithm is trained it may be that no further device data is provided to the fog computing device 103. However, especially where the fog computing device 103 periodically updates the training of the machine learning algorithm, the supply of device data from the edge computing device 102 to the fog computing device 103 may be an ongoing process.

[00120] It will be appreciated that the arrangement illustrated in figure 1 whereby for monitoring or control of a single building system the associated devices 101 communicate only with the respective edge computing device 102, which in turn forwards device data to the fog computing device advantageously provides a simple network topology. This is readily extendable to additional building systems whereby the fog computing device 10 need only communicate with a relatively small number of edge computing devices. However, there is no restriction to this particular network topology. The or each device 101 may also communicate its device data directly to the fog computing device 103 in addition to the edge computing device 102. Optionally, in some examples or for some devices, device data may be sent to the fog computing device 103 and forwarded to the edge computing device 102 as required. In some embodiments there may be a blend by which some devices 101 communicate primarily with the edge computing device and some devices 102 communicate primarily with the fog computing device 103, or both (an example of which is presented in figure 3).

[00121] In one embodiment data collection is implemented by the edge computing device 102 whereby device data is sent to the edge computing device 102 until a given threshold of data records is reached, at which point a data transfer to the fog computing device 103 occurs and training may commence.

[00122] Once a sufficient training data set of device data is built up at the fog computing device 103 (not specifically illustrated in figure 1) then machine learning or deep learning techniques are applied by the fog computing device 103 at step 107 to train or generate a trained machine learning algorithm. The process of training the machine learning algorithm is described in greater detail below in connection with figures 8 to 10. It will be appreciated that the volume of training data required may be specific to the type of building system, the particular deployment scenario or the selected machine learning algorithm. Once the model is trained, it is provided to the edge computing device 102 at step 108 for processing device data.

[00123] The received trained model is executed at step 109. The executing trained Al model receives and process device data received at steps 104 from the or each device 101. As an example, a device 101 may be configured to periodically send device data to the edge computing device 102 according to a duty cycle or as new data becomes available (or data may be buffered and transmitted periodically). The executing trained Al model uses newly received device data as well as, optionally, previously received device data, which may also include the training set of device data noted above. Outputs from the executing trained Al model including control signals as appropriate are output at step 110. Execution of the model at step 109 will be dependent on the nature of the selected and trained machine learning algorithm, as will be apparent to the skilled person. The model output 110 may comprise an output message or alert to a user indicating a state of the building or the building system or if the state or the building or the building system state is outside of a defined range. Alternatively, or additionally, the model output 110 may comprise a prediction of a current or future state for of the building or the building system. Alternatively, or additionally, the model output 110 may comprise a signal instructing a component setting or change of operation for a component forming part of the building system either immediately or a future point in time. The component may comprise one of the devices 101 that supplies device data to the edge computing device for processing using the trained model, or it may be a separate device. A model output may be provided through a local area network portal, a mobile device application or a display or touchscreen device within the building.

[00124] As hinted at above, and indicated by the dashed lines, in some cases device data may continue to be supplied at step 111 to the fog computing device 103 (indicated as via the edge computing device 102, though it could be direct) for updating the training of the machine learning algorithm at step 112. The updated training may be continuous or periodic. The model update may take account of the training set of device data, previously received device data and / or newly received device data as appropriate. The updated model is supplied to the edge computing device 102 at step 113 which may then substitute the updated model for the previously received model for processing further device data (not illustrated).

[00125] An advantage of certain embodiments is that the edge computing device can execute a predictive machine learning and / or an inference machine learning algorithm without relying on cloud computing, with all the associated disadvantages that entails, as set out above. Furthermore, while only a single edge computing device associated with a single building system is illustrated in figure 1, the solution is scalable in that a relatively low cost edge computing device may be associated with each building system and executing a respective trained machine learning algorithm, while a single (or smaller number) fog computing device may perform the more computationally expensive but less frequently required task of training the respective machine learning algorithms. In a test system established by the present inventor, an edge computing device was implemented on hardware costing under £100, while a fog computing device was implemented on hardware costing around £500 (2022 prices).

[00126] The range of device data that may be collected by an edge computing device may be extended to enable context awareness and situation awareness. In particular, sensors or other data sources capable of providing an indication of building occupancy (or the whereabouts of normal building occupants) greatly enhance the ability of a building system to be responsive to the needs of the building’s users.

[00127] Context awareness refers to a system’s ability to understand the context in which it is operating, such as the user’s location, preferences, and behaviour, as well as the state of the system and its environment. The goal of context awareness is to provide users with a more personalized and relevant experience, by adapting to their needs and preferences. Situation awareness, on the other hand, refers to a system’s ability to understand the current situation or environment in which it is operating. This includes being aware of the current state of the system and its environment, as well as any potential threats or hazards. As an example, an air quality monitoring and control system implemented according to an embodiment may take account of the number (and optionally identity) of predicted building occupants when determining whether a predicted air quality measure is acceptable or whether it must be changed by control of ventilation system components.

[00128] Further examples of context awareness or situational awareness could include learning behavioural usage patterns of simple appliances like lighting, music, heating, ventilation, hot water, and even garden sprinklers. These behavioural models are not controlling the devices directly, rather the devices are observers, with habits learnt and improvements to the ecosystem proposed to other inference models. A further example could include computer vision inference being used to extend security or support a healthcare requirement like stopping someone falling down the stairs in an inference situational awareness model. Situational awareness involves not only understanding the behaviours and environment but also the situations itself. This could be a burglar breaking in with intent to harm, a potential fire, someone at harm doing DIY in the garden on a ladder that is not positioned correctly.

[00129] Referring now to figure 2, this is a process flow diagram for an embodiment comprising a modification to the distributed Al architecture of figure 1 in which both a predictive machine learning model and an inference machine learning model are executed upon the edge computing device 102. For figure 2, and also for figures 3 to 6 discussed below, where steps are the same as presented and described for an earlier figure, the same reference numbers are used, and the description is not repeated.

[00130] Figure 2 differs from figure 1 in that on receipt of the device data at step 106 the fog computing device 103 trains two machine learning algorithms to generate two trained models at step 201. Both a trained predictive model and a trained inference model are supplied to the edge computing device 102 at steps 202 and 203. Both trained models are executed by the edge computing device 102 at step 204 and one or both models may generate model outputs at step 205.

[00131] Particularly, the predictive machine learning model may be adapted to provide a prediction of a current or future state for the building or the building system. The inference machine learning model may be adapted to provide a control signal instructing a component setting or a change of operation for a component forming part of the building system either immediately or a future point in time to control or adjust a predicted state for the building system. In some embodiments the prediction from the predictive machine learning algorithm may comprise an input to the inference machine learning algorithm (optionally, along with device data relating to sensors or system components). The inference machine learning model allows for the identification of system components that influence a building or building system state and which may be controlled to adjust a prediction of a future state. Advantageously, the distributed Al architecture of figure 2 allows both a prediction of a current or future system state to be generated (which may or may not be provided to a user or building occupant) and the building system to be controlled to adjust the system state, for instance if the predicted system state falls outside of a predetermined range.

[00132] Referring now to figure 3, this is a process flow diagram for an embodiment comprising a modification to the distributed Al architecture of figure 1 in which both sensor data and data from a component 301 forming part of a building control system are processed to train a machine learning algorithm (or processed when the trained model is executed, or both) for monitoring or controlling a building system.

[00133] Accordingly, figure 3 illustrates at least one (optionally, plural as indicated by the dashed lines) sensors 101 providing sensor data to the edge computing device 102 at step 104 and at least one (optionally, plural as indicated by the dashed lines) system components 301 providing system component state data to the fog computing device 103 at step 302. Although not explicitly illustrated, in some embodiments the fog computing device 103 may provide the system component data to the edge computing device in an analogous way to the supply of sensor data from the edge computing device 102 to the fog computing device 103 at step 106. A further alternative is for the system components 301 to communicate systems component data to both the fog computing device 103 and the edge computing device 102, or primarily to the edge computing device. Optionally, the system component data may be stored by the fog computing device 103 at step 303. More generally, either of the edge computing device 102 and the fog computing device 103 or both may store any portion of data from the sensors 101 and the system components 301 for use in model training or model execution. Irrespective of the network topology, at step 107 the training of the machine learning algorithm may be based on a training data set comprising data from both sensors 101 and system components 301.

[00134] At step 109, as before the trained Al model processes data received from the devices 101 at step 104. Additionally, the trained Al model may process data received from the system components 301 at step 302 (either directly or via the fog computing device 103). That is, both the devices 101 and the system components 301 send data to the edge computing device 102 that is processed using the trained machine learning model at step 109 to generate model outputs.

[00135] At step 110, as before the trained machine learning model generates model outputs. Those outputs may comprise one or more control signals for controlling one or more of the system components 301. That control may be responsive to a prediction of a current or future system state.

[00136] Although not specifically illustrated, updating of the trained model at step 112 may also take account of further received system component data.

[00137] Figure 4 is a further refinement to the process flow diagram of figure 3 in which the fog computing device 103 trains both a predictive machine learning model and an inference machine learning model, as for figure 2, on the basis of both sensor data and system component data. As for figure 2, both trained models are provided to the edge computing device which may execute both models, processing received sensor data and / or system component data to provide model outputs (which may include control signals or data controlling the system components 301).

[00138] Figure 5 is a process flow diagram for an embodiment comprising a modification to the distributed Al architecture of figure 4 in which the distributed Al architecture further comprises both a sensing edge computing device 501 which executes a predictive machine learning model and a system edge computing device 502 which executes an inference machine learning model. The edge computing devices 501,502 both form part of a distributed Al architecture for monitoring or controlling a single building system. Each edge computing device 501,502 may process both sensor data from sensors 101 and system component data from a components 301 forming part of a building control system. More generally, it may be that each edge computing device 501,502 processes part or the whole of the available sensor and component data.

[00139] For clarity, in figure 5 (and also figure 6) the processing of updating the training of the machine learning models is not illustrated, though the skilled reader will appreciate that this process is analogous to that illustrated for figure 4.

[00140] As for figure 4 the distributed Al architecture includes a single fog computing device 103 which is responsible for machine learning algorithm training to generate both predictive and inference machine learning models at step 201, using part or the whole of the available sensor and component data. However, differing from figure 4, at step 503 the trained predictive model is supplied to the sensing edge computing device 501 and the trained inference model is supplied to the system edge computing device 502 at step 504.

[00141] Figure 5 illustrates an example embodiment in which the sensing edge computing device executes the predictive model at step 505 processing sensor data received from one or more sensors 101. The supply of device data to the executing predictive model at step 505 is not illustrated for clarity in figure 5. At step 506 the predictive model generates a model output, for instance a prediction of a current or future state of the building or the controlled building system based on the sensor data. However, in certain other embodiments the predictive model may also accept as inputs data from one or more system components 301. The output at step 506 may be a value for a current or future system state. Alternatively, the output may be an indication that a predicted value exceeds a threshold or falls outside of a predefined range. At step 507 either an indication that a threshold or range will be breached, or a predicted current or future value of the building system state (or both) is transmitted from the sensing edge 501 to the system edge 502 where it forms an input to the executing inference machine learning model at step 508.

[00142] Figure 5 illustrates an example embodiment in which the system edge computing device 502 executes the inference model at step 508 processing system component data received from one or more system components 301. The supply of system component state data to the executing inference model at step 508 is not specifically illustrated for clarity. However, in certain other embodiments the inference model may also accept as inputs data from one or more sensors 101. Furthermore, the inference model receives as an input a prediction of a system state or an indication that this falls outside of a predefined range from the predictive model executing on the sensing edge 501. The executing inference model generates an output at step 509. The outputs from the inference model comprise control data 510 that is supplied to system components 301 at step 510.

[00143] The control data output 510 comprises settings or instructions for control of building system components 301 to maintain the state of the system within normal operational bounds. The provision of a separate system edge computing device 502 allows for closer integration between the control of the building system and the building system components 301. Such improved control may include Digital Signal Processing (DSP) based control of system components, for instance a ventilation fan, as opposed to conventional digital control whereby components are typically only switched on or off.

[00144] Furthermore, where a prediction or indication that a prediction falls outside of a defined range is received by the inference model at step 508, the close integration between the system edge and the system components allows for more flexible control of the system components 301 to adjust the system to ensure that the system state remains in a predefined range. An example of this is presented in figure 6. Figure 6 is like figure 5 for the specific example of an air quality monitoring and control system based on the same distributed Al architecture of figure 5. Sensors 101 are now identified specifically as environment sensors, for instance temperature, pressure, humidity, particulate matter, and certain gases. This sensor data is processed by an environment sensing edge 501. The building system is specifically a ventilation system. In one example, this may be an MVHR system, which may be referred to as a “smart MVHR system” through its integration with a ventilation system edge 502. The ventilation system components 301 may include fans, valves, heaters, and other conventional components of a ventilation system. The ventilation system components 301 may be configured to provide data indicative of a setting or state such as a fan speed, airflow or whether a valve is open, and to receive control data to control those states.

[00145] At step 505 the environment sensing edge executes a predictive model based on sensor data (and optionally state data for components of the ventilation system). It forms a prediction at step 506 fora current or future measure of air quality for the building. If this falls outside of accepted bounds then at step 507 an output is passed to the inference model executing on the smart MVHR, which is adapted to adjust the operation of the ventilation system components so as to prevent the air quality falling outside of the accepted bounds. As an example, if it is predicted that air quality will reduce at a future time, then in advance of that time point the ventilation system may be adjusted to increase the supply of fresh air into the home.

[00146] Although not specifically illustrated in figure 6, as well as sensor data the predictive model may also receive data from other sources. For instance, occupancy sensors, for instance cameras may detect how many people (and optionally, which particular people) are present in a home at a given time. Through observation of typical occupancy patterns the training of the predictive model may take account of how many people and which people are typically present within a home at different times of day and days of the week. This may be correlated during the training process with observed fluctuations in air quality which may correspond to activities within the home, for instance showering or cooking at "normal" times of day. This information about occupancy enables a degree of situational awareness to be built into the building system monitoring and control: a prediction of a future reduction in air quality may result from a prediction that based on observed behavioural patterns for building occupants it is likely that the activities of the building occupants within the home will lead to a drop in air quality at a future time.

[00147] Furthermore, the operation of the inference model may be optimised to not only address a predicted drop in air quality, but to do so in an efficient manner (for instance, adjusting heating or ventilation components such that they operate in advance of the predicted low air quality during times where energy prices are lower or where electricity is predominantly supplied by low carbon sources.

[00148] In certain embodiments the predictive model may be further trained and respond to other data sources. For instance, particularly for the example of an air quality monitoring and control system, weather forecast data may be received through a network connection (as described below in connection with figure 13), which may affect how the ventilation system is to be controlled to remain within acceptable bounds. Information received from outside of the home may also include occupancy information, for instance mobile device location data indicating the whereabouts of frequent building occupants and calendar information indicating the likely future whereabouts of frequent building occupants.

[00149] Referring now to figures 7a and 7b, as noted previously, while embodiments of the present invention are applicable to buildings in general, they are particularly to suited to domestic premises such as houses and apartments (referred to generically in this document as homes). Partly this is because the daily routines of people within a home and the range of activities that take place within a home are significantly more complex than for industrial, retail or commercial premises, with the result that the ability to implement situational awareness functions described above taking account of building occupancy are especially beneficial. Figure 7a is a perspective view of one particular form of home, and figure 7b illustrates a cross section plan view of such a home incorporating a distributed Al architecture for monitoring or controlling air quality.

[00150] Figures 7a and 7b, show a home 700 which is constructed as a hybrid building 700 (which may also be referred to as a modular building). The hybrid building 700 comprises a first building section 701 and a second building section 702. The first building section 701 can take the form of a dock. The term “dock” is used to describe a building section to which another building section may be docked, installed, connected, or attached. The first building section 701 is an on-site construction at a final location of the building 700, which may be a fixed, on-site location. That is, the building 700 has a final, fixed, on-site location. In this exemplary embodiment, said final, fixed, on-site location is determined by construction plans and fixed by virtue of building foundations. The term “on-site location” is used to refer to the building site, which will be understood to refer to the immediate proximity of the building 700 and the entire building site, including housing estate, on which the building 700 is to be built. It will be understood that the site may be a large building / construction site comprising a plurality of plots, the final location for the building 100 being provided by one of said plots.

[00151] The second building section 702 can take the form of a module. The term “module" is used to describe a building section which may be docked to another building section or is installable, connectable, or attachable to another building section, particularly a first building section in the form of a dock. The second building section 702 is transportable to the final location in a substantially assembled form.

[00152] During construction of the first building section 701 at the on-site location, the first building section 701 is preconfigured to receive the second building section 702. That is, the first building section 702 is constructed with the knowledge and design that a second building section is to be subsequently connected, and the first building section 701 is thus preconfigured for connection and receipt of the second building section. The second building section 702 may be removable. This could relate to the shaping of the first building section 701, through to sealing and connection features. The second building section 702 is connected to the first building section 701 at the final location of the building 700, thereby to provide said building 700 at the final, fixed, on-site location.

[00153] The first building section 701 further includes a services hub 703, which may include part or the whole of certain building systems. For instance, for a ventilation system, the services hub 703 may include ventilation system components such as the pumping and heat exchange elements of an MVHR.

[00154] Figure 7b shows the system edge computing device 502 and the fog computing device 103 located within the services hub along within at least one system component 301, though it will be understood that this location is merely exemplary, and the system edge 502 and the fog computing device 103 may be located at any convenient location. Similarly, the sensing edge computing device 501 is shown located within the first building section 701, though it may be located at any convenient location. Figure 7b shows a range of sensors 101 and system components 301 distributed through the first and second building sections 701,702. It will be understood that their locations are determined according either to optimal placement for sensing or where required to form an effective part of the building system. The sensors 101, system components 301, fog computing device 103 and edge computing devices 501,502 may be connected wirelessly, as will be described in greater detail below in connection with figure 13.

[00155] Referring now to figure 8, this is a flow chart illustrating a process of training and implementing a machine learning algorithm for monitoring or controlling a building system where a distributed Al architecture is deployed within a building. As an example, this may relate to the air quality monitoring and control of figure 6. Figure 8 specifically relates to an example in which a machine learning model for monitoring and controlling air quality has been previously established but is to be newly deployed within a building such as the hybrid home of figures 7a and 7b. It will be understood that the behaviour of a specific building regarding patterns of air quality must first be observed during a training process.

[00156] At step 801 the fog computing device 103 collects device data relating to the environment sensors 101 and the ventilation system components 301. As previously described, this data may be received via the sensing or system edge computing devices 501,502 or directly from the sensors 101 or ventilation system components 301. Data is collected until a sufficiently large training data set is established. The size of that data set may be dependent on a particular building, but for instance may comprise a period of several days, weeks or months. It will be understood that during that time the ventilation system may be controlled by the ventilation system edge 502 according to a default or less fully featured control algorithm.

[00157] At step 802 once the training data set is established then the machine learning algorithm or algorithms may be trained, for instance using machine learning or specifically deep learning techniques. This may include, for instance, Exploratory Data Analysis (EDA), clustering machine learning techniques such k-means for correlation matching, machine learning gradient descent fit regression algorithms and deep learning Long Short-Term Memory (LSTM) to generate trained predictive and inference models. This training process is described in greater detail in connection with figure 10, below. The trained model or each of a predictive model and an inference model are supplied to the edge computing device 102 or respective edge computing devices 501,502.

[00158] At step 803 the edge computing device 102 or each of the edge computing devices 501, 502 receive device data from sensors and / or system components and at step 804 the edge computing devices 501,502 implement the trained models. Appropriate model outputs such as alerts, or control signals are generated at step 805.

[00159] Figure 9 is a modification to the flow chart of figure 8 further illustrating a method of updating the training of a machine learning algorithm for monitoring or controlling a building system according to an embodiment. Specifically, figure 9 illustrates a feedback loop whereby in addition to the collected device data at step 803 being supplied to the edge computing device for implementing the machine learning algorithm at step 804, it is further supplied to the fog computing device 103 for updating the machine learning algorithm at step 901. Once an updated machine learning algorithm is generated then the updated algorithm is implemented at step 804.

[00160] Figure 10 is a modification to the flow chart of figure 8 further illustrating a more detailed method of training of a machine learning algorithm for monitoring or controlling a building system according to an embodiment. The training method of figure 10 may be applicable to training a machine learning model for the first time, though it will be understood that part or the whole of the method of figure 10 may be applicable when deploying a previously identified model algorithm for the first time in a new building.

[00161] As for figures 8 and 9, the method of figure 10 begins with the collection of device data at step 801. Particularly where a new type of building system is considered for model development, it may be that a considerable period of data collection may be required to generate a training data set for model development, for instance over a period of days, weeks or months. It may be that the data collection phase comprises collecting data from a large number of devices including sensors and system components, whereas during model development it may be that some data sources may be excluded as not being significant influences on the building system or not generating significant data for the generation of model outputs such as predictions and control signals. Step 801 may further include removing missing values, handling outliers, normalizing the data, and encoding categorical variables.

[00162] At step 1001 algorithm type may be evaluated. This may be a largely manual process, or it may be at least partially automated through analysis of the device data and an understanding of type of monitoring or control outputs that are required. Essentially step 1001 comprises selection of a suitable machine learning algorithm for us in the building system. It may be that more than one algorithm is evaluated and tested, or trained, before the most suited algorithm is selected. Particularly, deep learning algorithms based on exploitation of a neural network may be evaluated and selected for model development. The skilled person will be aware of suitable machine learning algorithms that may be deployed. However, suitable examples include linear regression, logistic regression, decision tree, SVM algorithm, naive Bayes algorithm, KNN algorithm, K-means, random forest algorithm, dimensionality reduction algorithms, gradient boosting algorithms, gradient descent fit algorithms, Long Short-Term Memory (LSTM) and the AdaBoost algorithm.

[00163] At step 1002 data types are prepared. This comprises a process of identifying the data sources from the collected device data at step 801, identifying the data format including the precision and update frequency of the collected data, and defining data types to be provided to the algorithm type or types evaluated at step 1001. Step 1001 may further include transforming the data into a format that can be used by the machine learning algorithm. This may involve removing missing values, handling outliers, normalizing the data, and encoding categorical variables.

[00164] At step 1003 Exploratory Data Analysis (EDA) is performed based on the collected device data. Exploratory data analysis (EDA) in machine learning refers to the process of analysing and understanding data to extract insights, patterns, and trends. During EDA, the data is analysed for missing values, outliers, skewness, and other data quality issues. Data distributions and correlations are explored using summary statistics and visualization techniques, such as histograms, box plots, scatter plots, and heatmaps. EDA also involves the identification of potential features and the formulation of hypotheses that can be tested through machine learning models. EDA is an essential step in machine learning because it helps to identify the distribution and characteristics of the data, understand relationships between variables, detect outliers and anomalies, and ultimately prepare data for modelling. EDA typically involves techniques such as data visualization, statistical analysis, and data summarization. The goal of EDA is to gain a better understanding of the data and inform decisions about feature engineering, model selection, and other important aspects of the machine learning process. Ultimately, EDA helps to ensure that machine learning models are trained on high-quality data, which is critical for achieving accurate and reliable predictions.

[00165] At step 1004 the machine learning algorithm is built, taking account of the model selection and prepared data types from the preceding steps.

[00166] At step 802 the machine learning algorithm is trained. Training the model involves using the data to adjust the model parameters so that it can accurately make predictions or decisions. For instance, this may involve defining a loss function that measures the difference between the model predictions and the actual outcomes, and using an optimization algorithm to minimize this loss function. At step 1005 the machine learning algorithm is evaluated. Evaluation may involve testing the performance of the trained model on a separate dataset to determine how well it generalizes to new data. This may require the collection of further data, or it may be that the data collected at step 801 is partitioned into two parts, a first part for model training and a second part for model evaluation. This may involve computing various performance metrics, such as accuracy, precision, recall, and F1 -score.

[00167] At step 1006 the model may be fine-tuned, for instance if certain data sources or data types are found to not be influential in monitoring or controlling the building system they may be excluded. Or it may be that model prediction accuracy indicates that further data sources may be required. As an example, for a system modelling air quality within a home it may be identified that more environment sensors covering particular portions of the home may be required.

[00168] At step 1007 the model is deployed. For instance, this may comprise the step of supplying the trained model to an edge computing device as described above in connection with figures 1 to 6.

[00169] Turning now to figure 11, this is a flow chart illustrating a method of operating an edge computing device for monitoring or controlling a building system according to an embodiment. In certain embodiments, the edge computing device serves to receive some or all the device data at step 1101, optionally store the device data step 1102 and supply the device data to the fog computing device at step 1103 for model training. However, as noted above, while this arrangement is logical and readily scalable for other building systems, if may be that some or all the devices supply data also or primarily to the fog computing device.

[00170] At step 1104 the edge computing device receives one or more trained machine learning algorithms from the fog computing device for monitoring or controlling a building system. At step 1106 the trained algorithm is executed for processing device data received at step 1105, and model outputs are generated at step 1107. Optionally, though not illustrated in figure 11, device data received at step 1105 may be supplied to the fog computing device for updating the training.

[00171] Turning now to figure 12, this is a flow chart illustrating a method of operating a fog computing device fortraining a machine learning algorithm for monitoring or controlling a building system according to an embodiment. Device data is received at step 1201, either via the edge computing device or directly (or a combination of both). At step 1202 the machined learning algorithm is trained, for instance according to the method of figure 10. At step 1203 the trained algorithm is supplied to the edge computing device for implementation.

[00172] At step 1204 the fog computing device receives further device data which is used for updating the training at step 1205 such that an updated trained algorithm may be supplied to the edge computing device at step 1206.

[00173] Figure 13 is a network topology diagram fora distributed Al architecture according to an embodiment. Specifically, figure 13 is a network diagram showing a distributed Al architecture within a building such as a home. A single fog computing device 103 is illustrated. This is communicating with a sensing edge 501 and a system edge 502 according to the embodiment of figure 5, (for instance, for monitoring or controlling a building ventilation system as illustrated in figure 6). However, in some examples there may only be a single edge computing device 102 for controlling a building system (for instance according to the embodiments of figures 1 to 4).

[00174] Furthermore, as previously described, a key advantage of certain embodiments is that it is scalable for controlling multiple building systems within a building, for instance a home, with a single fog computing device (or a low number of fog computing devices) providing machine learning algorithm training and providing those trained models to an array of edge computing devices associated with the multiple building systems. Each building system may be associated with one or more edge computing devices. In some cases, an edge computing device may execute trained machine learning models for more than one building system, particularly where there is overlap in the devices that provide data to be processed by those models, as discussed below. Accordingly, figure 13 illustrates this scalability by the representation of additional sensing edge computing devices 501 or system edge computing devices 502 indicated by dashed lines. Furthermore, the fog computing device 13 provides further orchestration and coordination services for the control of the multiple building systems.

[00175] Figure 13 further illustrates a variety of devices that provide device data for training a machine learning model or to be processed by a trained model. For instance, there may be at least one sensor 101 (and optionally multiple sensors as indicated by the dashed lines). Alternatively, or in addition, there may be at least one building system component 503 and optionally multiple building system components as indicated by the dashed lines). In some embodiments it may be that each sensor 101 or system component 503 is associated only with a single building system that is monitored or controlled by a single edge computing device 501,502. However, in other embodiments I may be that at least one sensor 101 or system component 503 is associated more than one building system, and those building systems may be monitored or controlled by one or more edge computing device 501,502. It will be understood that the arrangement of sensors 101, system components 503 and edge computing devices 501,502 will be context dependent for the particular building and building systems to be monitored or controlled. In particular, the precise way in which device data is communicated through the distributed Al architecture will be implementation specific.

[00176] The sensors 101, system components 503, edge computing devices 501,502 and fog computing device 103 communicate with one another through a building network 1301. This may a wired or a wireless network. There is no limitation to the nature of the network, nor the communication protocols used to communicate across the network. However, as one example, the Matter protocol, formerly known as Project CHIP (Connected Home over IP), is an open-source standard for smart home devices which may be used in some embodiments, at least for communicating device data from sensors 101 and system components 503 and optionally control data for the system components 503. The Matter protocol specification is defined by the Connectivity Standards Alliance and is available at the following URL: The goal of the Matter protocol is to create a unified, secure, and reliable standard for smart home devices, so that they can work seamlessly with each other and with different ecosystems. In certain examples the edge computing device and the fog computing device may be Matter compliant so as to be able to receive device data from standard Matter compatible smart home sensors and system components. In some cases, off-the-shelf Matter compatible home devices may be used to form sensors or system components, with the edge computing device or the fog computing device configured to receive the communicated information in addition to or in place of a conventional data end point forthose off-the-shelf components.

[00177] The interoperability Matter protocol is designed to work with a variety of wired and wireless technologies, including Wi-Fi, Bluetooth Low Energy, and Thread. That is, the building network 1301 may suitably be any conventional wireless network. The Matter protocol defines a set of common communication physical layer protocols, security mechanisms, and different vendor device profiles that can be used by smart home devices to communicate with each other within a single ecosystem.

[00178] Thread is a low-power, wireless networking protocol designed for smart home and Internet of Things (loT) devices. Thread is based on the IEEE 802.15.4 standard for low-rate Wireless Personal Area Networks (WPANs). The Thread specification is defined by the Thread Group and is available at the following URL: https : / / www.threadgroup,qrg / support#spec$^^ It uses a mesh networking architecture, which allows devices to communicate with each other directly or through other devices in the network. This means that Thread can provide a reliable and flexible network for smart home devices, even in environments with many devices and obstacles.

[00179] One of the key advantages of Thread is its low power consumption. Thread-enabled devices can operate on small batteries for months or even years, depending on the usage. This makes it ideal for smart home devices such as sensors and switches, which need to operate for long periods of time without requiring frequent battery replacements.

[00180] Figure 13 further illustrates a router 1302 coupled to the building network 1301. In some embodiments the router may control the building network 1301, for instance where the building network is a Wi-Fi network and the router 1302 acts as a Wi-Fi access point. The router 1302 links the distributed Al architecture to the internet, and particularly to cloud computing services indicated by cloud 1303. In some embodiments the fog computing device 103 may be combined with or implement the functionality of a router 1302 such that no separate router 1302 is required.

[00181] According to certain embodiments described above an edge computing device and a fog computing device comprise a distributed Al architecture for monitoring or controlling building systems such that building systems may be controlled more effectively without recourse to cloud computing to execute Al algorithms. That is, the edge computing layer and the fog computing layer do not require there to be an available cloud computing layer to implement the above embodiments. However, there may be embodiments in which coordination with cloud computing services may be desirable. As an example, data from building systems may be collected and stored in the cloud 1303. Data from multiple buildings (each implementing their own distributed Al architecture) may be aggregated which may allow for greater insights to be gathered in how building systems operate. This may in turn guide the development of enhanced building control algorithms in the future.

[00182] Furthermore, the presence of a router 1302 and access to cloud computing services allows for additional data to be collected for use in training and executing building control Al models. As noted previously, for monitoring or controlling a ventilation system it may be desirable to obtain information about forecast weather or occupancy information from an outside data source.

[00183] Figure 14 illustrates an edge computing device 102 (or 501,502) forming part of a distributed Al architecture. The edge computing device comprises a processor 1401, memory 1402 and some form of input / output device 1403 configured to communicate with other devices across a network, for instance the building network 1301 of figure 13. The memory 1402 stores executable instructions that, in response to execution by the processor 1401, cause the edge computing device to perform the method described above in connection with figure 11 or any of the other functions described above in connection with the other embodiments. The input / output device 1403 allows the edge computing device to communicate with devices such as sensors 101 and system components 503 for receiving device data and transmitting control data, as well as to communicate with the fog computing device 103. In particular, the edge computing device receives a trained Al model (or an update to the trained Al model) from the fog computing device, which is stored in the memory 1402. The processor 1401 executes the trained Al model and uses this to process device data.

[00184] According to certain embodiments, the edge computing device may be a relatively low cost computing device so long as it has the computational capacity to executed a trained Al model. For instance, for monitoring or controlling some building systems a Raspberry Pi computing device may be used, and is available from Raspberry Pi Ltd and described at the following URL: https: / / www.raspberi~ypi.com / . In some embodiments, advantageously the edge computing device incorporates hardware support for predictive or inference Al acceleration.

[00185] Figure 15 illustrates a fog computing device 103 forming part of a distributed Al architecture according to an embodiment. The fog computing device comprises a processor 1501, memory 1502 and some form of input / output device 1503 configured to communicate with other devices across a network, for instance the building network 1301 of figure 13. The memory 1502 stores executable instructions that, in response to execution by the processor 1501, cause the fog computing device to perform the method described above in connection with figure 12 or any of the other functions described above in connection with the other embodiments (particularly in relation to training a machine learning algorithm as described in connection with figures 8 to 10). The input / output device 1503 allows the fog computing device to communicate with devices, particularly edge computing devices 102, 501,502 and optionally communicate directly with sensors 101 and system components 503. In particular, the fog computing device receives device data to form a training data set and uses this to train a machine learning algorithm to then provide a trained Al model (or an update to the trained Al model) to an edge computing device.

[00186] According to certain embodiments, the fog computing device may be a relatively expensive computing device (relative to an edge computing device) in order to provide the computational capacity to train a machine learning algorithm in support of an edge computing device to execute the trained algorithm. For instance, suitably an NVIDIA Xavier computing device may be used, which is available from Nvidia Corporation and described at the following URL: https: / / www.nvidia.corn / en-qb / au machines / embedded-systems / ietson-xavier-nx / . Nvidia Xavier computing devices provide for hardware Al acceleration allowing for machine learning algorithm training within the building. In some embodiments, Al acceleration (and in particular, Al inference acceleration) may be provided by leveraging multiple Graphics Processing Units (GPUs) within the fog computing device in parallel.

[00187] In certain embodiments, for at least one building system where a plurality of building control systems is monitored or controlled it may be that in addition to the fog computing device training a machine learning algorithm adapted for monitoring or controlling that system, the fog computing device also executes the trained Al model to process received device data. While a key advantage is scalability through the separation of training and prediction or inference execution, in some situations it may be desirable for the fog computing device to additionally be responsible for model execution.

[00188] In certain embodiments, in addition to their functions described above, at least one edge computing device or fog computing device may receive device data from at least one controllable device within or associated with the building, process the device data using a control algorithm, and return a control signal to the controllable device. In particular, while the above described computing architecture is suitable for controlling complex building system, including those with multiple sensor inputs and system components that influence system states, there may be relatively simple systems or single device that require a degree of control but may not require the use of Al. Spare computing capacity upon one of the devices within the distributed Al architecture may suitably be used to achieve that control. For instance, a relatively simple device that may require control may include a kitchen appliance, for instance an oven, microwave, refrigerator, freezer or dishwasher; a utility appliance, for instance a clothes washing machine or tumble drier; or a home entertainment device, for instance a television or other audio or visual device.

[00189] Figure 16 illustrates a building or building system sensor device 101 configured to communicate sensor data to a distributed Al architecture. At a minimum the sensor device 101 comprises a sensor 1601 capable of generating sensor data indicative of at least one sensed parameter and some form of input / output device 1602 configured to communicate that sensor data to other devices (particularly, edge computing devices) across a network, for instance the building network 1301 of figure 13. The sensed parameter may be an air quality parameter, for instance temperature, humidity, CO2, and particulate matter (for instance, 2 nm, 2.5 or 10 nm particulate matter). It will be appreciated that in other embodiments the sensor device 101 may include further components, for instance memory configured to buffer sensor data for periodic transmission and a processor to control the other parts.

[00190] Figure 17 illustrates a building system component 301 configured to communicate component data to a distributed Al architecture. At a minimum the building system component 301 comprises an actuator 1701 capable of providing system data indicative of an actuator state and some form of input / output device 1702 configured to communicate that system data to other devices (particularly, edge computing devices) across a network, for instance the building network 1301 of figure 13. The input / output device 1702 may further receive control data to control the actuator 1701. The actuator may be part of a ventilation system, for instance a fan motor, such that system data provided by the action 1701 may be fan speed and the control data may instruct a change in fan speed. It will be appreciated that in other embodiments the building system component 301 may include further components, for instance memory configured to buffer system data for periodic transmission and a processor to control the actuator in response to received control data.

[00191] Turning now to Figure 18, this is a process flow diagram for an embodiment corresponding to a modified form of the distributed Al architecture of figure 6 for the example of monitoring or controlling a ventilation system. Specifically, as noted previously, certain embodiments can provide context awareness and / or situation awareness as part of monitoring or controlling a building system. Figure 18 is one example of how this may be implemented.

[00192] Figure 18 corresponds to figure 6 except as noted below and those parts of the system and flows of information should be considered to be the same where indicated by the same reference number, and so will not be described.

[00193] Figure 18 differs from figure 6 firstly in that the ventilation system components 301 do not provide component state information 302 to the fog computing device 103 for training the Al models, though in other variants of the embodiment of figure 18 training of the Al models may continue to be based in part on component state data in addition to sensor data, for the purposes of predicting air quality and controlling the ventilation system components.

[00194] As for figure 6, a predictive Al model is trained by the fog computing device 103 and executed by the environment sensing edge 501 at step 505 for the purpose of forming predictions relating to air quality within the building. As for figure 6, an inference Al model is trained by the fog computing device 103 and executed by the system sensing edge 502 at step 509 for the purpose of controlling components of the ventilation system which the trained model has learnt influence the output of the predictive Al model. Specifically, as for figure 6, the inference model receives as an input a prediction from the predictive model, for instance a prediction that a predicted air quality will fall outside of an air quality or system state range (that is, corrective action is required). The inference model generates control data outputs 510 that are provided to components of the ventilation system that influence the predictive model output to seek to ensure that predicted air quality does not exceed the state range (for instance, to ensure that air quality stays within safe limits).

[00195] Figure 18 differs from figure 6 in that as well as training predictive and inference models based on sensor data (and optionally ventilation system component data), the fog computing device 103 further trains an observational Al model ora reinforcement learning model that is executed by the system edge 502.

[00196] An observational Al model is a type of machine learning model that is designed to learn patterns and relationships in data, in this context the behaviour of a ventilation system by observing and analysing it, without any direct intervention or feedback from humans. Observational Al models are often used fortasks such as anomaly detection, prediction, and classification, where the model is trained on large amounts of data to identify patterns or relationships that can be used to make accurate predictions or classifications on new data. These models are typically trained using unsupervised learning methods, where the model is left to identify patterns in the data without being explicitly told what to look for.

[00197] A reinforcement learning Al model is a type of machine learning model that learns to make decisions based on trial and error through interacting with an environment. It involves an agent that is trained to take actions in an environment to maximize a cumulative reward signal, which is a scalar value that measures the agent's success in achieving its goals. In reinforcement learning, the agent learns by receiving feedback in the form of a reward signal based on its actions. The agent tries to maximize the cumulative reward overtime by choosing actions that lead to higher rewards and avoiding actions that lead to lower rewards.

[00198] In the embodiment of figure 18 it is specifically a reinforcement learning model that is executed by the system edge computing device. The purpose of the reinforcement model is to observe and record the states and actions of components of the building system, in this context the ventilation system, and the states and actions of the building occupants. Specifically, at step 1801 observations of the control data sent to the ventilation system components (and optionally the ventilation system component state data) are stored and transmitted to the fog computing device at step 1802. In some cases, this information may not be stored first. As previously described, the fog computing device 103 is also in receipt of environment sensing data indicative of air quality, and also occupancy information relating to the building occupants.

[00199] The purpose of the reinforcement model is to determine what patterns exist within the data, not to control the components devices itself. For the specific example of figure 18 the observations used to train the reinforcement model in relation to air quality and a ventilation system may include: • What air quality threshold or range is set. • Current and historic air quality data. • Algorithm banding for the reinforcement model. • When the occupants are present. • How long building occupants are present. • Time of day that building occupants are present. • How long the inference model and the control data to the ventilation system takes to achieve a desired level. • How much power is needed.

[00200] These metrics are indexed with a machine learning algorithm, classified and a decision discovery machine learning or deep learning algorithm applied.

[00201] The trained reinforcement model is provided to the system edge 502 at step 1803 and executed at step 1804. At step 1805 the executing reinforcement model provides an input to the predictive model. This enables the reinforcement model not only to consider the raw metrics but also behavioural perceptions. An example of this is the reinforcement model adjusts the air quality threshold within safe limits of the banding autonomously without any human intervention if the occupant does not disagree with the change, then the change becomes the threshold.

[00202] The reinforcement model not only considers the raw metrics of the building system component usage but can adjust thresholds autonomously through behavioural perceptions with a condition of safety limits or threshold banding. In the event that the occupant does not raise a concern to the change, the change becomes the new threshold.

[00203] Advantageously, this additional use of a reinforcement model allows for habits and preferences of specific building occupants to be learnt without human intervention, which create historic datasets for future usage and prediction. This can help to reduce energy consumption due to learning new occupant thresholds, for instance for a comfortable heating range.

[00204] As previously described, there are multiple further ways in which the distributed computing platform described above may be effectively and efficiently used. This may take advantage of the fog computing device as required for computationally expensive processing, in combination with one or more edge computing device to perform other parts of the processing. That is, the single fog computing device described above may be used for a portion of its time on one task (such as the building system control algorithm training described above) and be used for another portion of its time on another task. An example relating to video processing will now be presented.

[00205] It is commonplace to deploy video cameras within a building such as a home, particularly as part of a security system. It is known for the cameras to implement a certain degree of video processing, for instance basic object detection, which may be used as a trigger to initiate recording of the video data at a Network Video Recorder (NVR) or to trigger an external system (such as a security alerting system). However, conventional video cameras implementing object detection often result in a large number of false positives (that is, an indication that a predetermined object type has been detected when it is not in fact present). This may cause the user to lose confidence in the accuracy of the security system.

[00206] A conventional security system will now be described in connection with figure 19. Figure 19 shows a plurality of video cameras 1901 (there may be any number including only a single camera). These may be located about a building such as home, for instance monitoring entrances to the building or the grounds around the building. The cameras may be conventional, off the shelf IP cameras. That is, they may communicate through the Internet Protocol (IP) protocol wirelessly or suitably through a wired connection. Figure 19 shows each camera connected through a wired connection to a Power Over Ethernet (POE) switch 1902, though any suitable communication technique and power supply may be used. In turn, the POE switch 1902 is connected to an NVR or Digital Video Recorder (DVR) 1903 for recording video data supplied by each camera 1901. The POE switch 1902 is also connected to an internet hub 1904 to enable communication between the NVR / DVR 1903 and an external service 1905 via a network such as the internet 1906.

[00207] The IP cameras 1901 send video to the NVR 1903, for instance using the Real Time Streaming Protocol (RTSP). Each camera 1901 also incorporates the native Al processing capability to perform basic object detection and feature extraction. Object detection notifications are communicated to the NVR 1903, for instance through the Open Network Video Interface Forum (ONVIF) protocol. For instance, as illustrated, each camera 1901 is capable of detecting a person, vehicle, or pet within the field of view and sending an object notification along with the underlying video data. For consistency with the embodiments of the invention described below, this may be referred to as layer A processing (though in figure 19 there is no layer B processing).

[00208] In response to an object detection notification the NVR 1903 decides whether to upload data to the cloud or send some form of notification, connecting via the switch 1902 and hub 1904. The NVR 1903 decides on the appropriate action. For instance, the external service may comprise a notification being sent to a user’s mobile device (illustrated at 1905).

[00209] As noted above, IP security cameras with some inbuilt Al capacity are known. They can perform object detection, for instance, to detect a person, animal, or vehicle and when so detected generate an output (for instance, an alert to a user’s device with a stream of the video - live - or a clip showing the detected object). However, the known problem is the high number of false positives - for instance, a leaf blowing past camera, a drop of water on lens etc. may give rise to a false object detection notification. Users may start to ignore alerts sent to their mobile devices 1905. A further issue is that conventional security camera systems have no consideration for “Context Awareness” of their surroundings.

[00210] One option to address the problem of false positives would be to increase the Al capability of cameras, however this is not considered practicable due to the high numbers of installed cameras, and the cost of providing additional computing capacity as frequently a large number of cameras are deployed in a building. An alternative approach would be for security camera video footage to be passed to the cloud for processing. However, this raises privacy concerns.

[00211] The present inventor has identified that for a home where a distributed computing architecture as described above is already employed then this may be leveraged to provide for improved Al driven smart home computer vision. This may be particularly suitable as part of a building security system. This provides the benefit of economy of scale, partly by leveraging the existing computing capacity of the fog computing device and partly by not requiring each camera to be equipped with more expensive computing capability - existing, low cost, off the shelf IP cameras may be used, performing their native object detection.

[00212] Turning now to figure 20, this illustrates a video processing system according to an embodiment of the present invention. In this context, the terms “video processing” or “video data processing” should not be understood as referring to conventional processes such as encoding and decoding. Rather, what is meant is a process whereby video data is processed using Al to perform inferences about the content of the video data. This may be referred to instead as “computer vision inference”. The present invention provides a hierarchical, scalable, computer vision architecture that accelerates computer vision inference. Where components are the same as for figure 19, the same reference number is used, and they will not be described again. In particular, the same form of IP camera, POE switch, and internet hub may be used, and communication may suitably take place through the RTSP and ONVIF protocols (however the present invention is not limited to any particular communication protocols).

[00213] The present invention is in essence a processing prioritisation system. As for the prior art of figure 19, all video data processed at each video camera 1901 to perform basic object detection, for instance to identify a person, vehicle, or pet. This may be referred to as Layer A processing.

[00214] If an object is detected, the result and the underlying video passed (for instance, via the POW switch 1902) to an edge processing device 2001 which performs further classification to reduce the incidence of false positives and to make judgements about what types of objects have been detected. This may be referred to as Layer B processing. The edge processing device 2001 may be generally the same as described earlier in this patent specification. In some cases, an edge processing device that is also responsible for monitoring a building system may also be used for video processing. In other cases, a dedicated edge processing device 2001 may be provided for video processing.

[00215] For example, where video data is processed as part of a security system, if a camera 1901 detects a person, the edge processing device may perform further classification (and / or image segmentation) to determine whether to classify the person as a child, teenager, or adult. Further classifications are possible, for instance whether the person is elderly. The edge processing device may also count the number of people present. Depending on the number or type or person detected, the edge processing device 2001 determines whether to pass the video data, object detection and result of Layer B processing to a fog processing device 2002 for further processing (Layer C processing). As well or instead an output may be generated by the edge computing device 2001, for instance, a conventional user alert as described above, which may be communicated by the internet hub 1904. Accordingly, data is only passed to the fog processing device 2002 when required, which is advantageous due to other computing demands placed on the fog computing device 2002.

[00216] As an example, if Layer B processing at the edge computing device 2001 detects multiple youths then the fog computing device 2002 may be used to perform Layer C processing to determine whether they represent a security threat. A security threat may be indicated by what the people are doing (for instance, climbing a wall), how dense the cloud is and whether they are carrying anything. The result of Layer C processing is that the fog computing device determines an appropriate response, which may for instance be to alert the user, inform police or private security, issue a visual or audible alarm or control any other part of a building system, for instance to lock exterior doors and windows.

[00217] In some examples, the prioritised processing system of the present invention is failed safe as the edge computing device can continue to perform processing if the fog computing device is unavailable. Similarly, the camera could revert to a conventional mode of operation if the edge computing device is unavailable.

[00218] In some examples, the fog is also responsible fortraining the classification model performed by the edge (with periodic transfer of classification data and underlying video from edge to fog), as described above. The fog may perform its own model training and execution.

[00219] Embodiments of the present invention described above operate in conjunction with a video camera with the inbuilt Al capability to perform object detection. According to other embodiments, the system may be adapted to work with “dumb” cameras that simply feed video data to the edge processing device to perform initial object detection (or at an intermediary object detection node). It will be understood that this may comprise a less scalable solutions than the use of IP cameras with native object detection, plus many of these are already deployed reducing user friction to buy into the system.

[00220] Other uses case scenarios include monitoring what a person eats for dietary reasons: the camera detects a person, the edge works out if they are eating, and the fog works out what they are eating. Also, the system may be used for elderly person monitoring: the camera detects a person, the edge works out if that person is elderly, and the fog monitors for alert conditions, for instance if the person is lying on floor for long time.

[00221] Embodiments of the present invention provide for context awareness: the presence of people is not necessarily a threat, but people somewhere they are not permitted can be a threat.

[00222] In some cases, the Layer B processing may be an extension of the Layer A processing at the camera. For instance, if camera is 32 point face recognition, the edge might be doing more detailed face recognition.

[00223] Turning now to figure 21, a computer implemented method of processing video data within a building according to an embodiment of the present invention is described in the form of a flowchart. At step 2101 a camera 1901 performs Layer A processing, for instance to perform object detection. At step 2101 a determination is made whether Layer A processing = true (for instance, if an object has been detected). If an object is detected, then a notification is transmitted to the edge computing device 2001 and Layer processing is performed at step 2103. The Layer B processing comprises further classification of the detected object or classification of a further object within the video data. At step 2104 the edge computing device 2001 determines whether Layer B processing of the video data is required. That is, whether the result of Layer B processing = true. If so, then at step 2105 the fog computing device 2002 performs Layer C processing. The Layer C processing determines whether to generate an output signal or a control signal for a building system based on the video data, the detected object, and the results of the first video data processing. If so (that is, Layer C processing = true then at step 2107 the fog computing device issues an output as described above.

[00224] The various processing layers will now be described in greater detail.

[00225] Layer A processing comprises principally object detection. This layer exists within the off the shelf IP camera itself which can perform object detection, light feature extraction and light classification. The most common being object detection typically performed to determine a person, vehicle, or pet. This level of implicit computer vision Al is low in computational requirements making it affordable for low powered embedded IP cameras. Suitably, Layer A processing may be executed using Haar Cascades, Principal Component Analysis (PCA), Convolutional Neural Networks (CNN), or a State Vector Machine (SVM).

[00226] Layer B processing comprises principally image segmentation, advanced classification, or model inference execution. This layer exists within the edge sub system and gets event triggered if Layer A = true. Layer B also performs computer vision segmentation and more computational classification beyond layer A capabilities. Segmentation partitions the image into multiple image segments this reduces the complexity of the scene and allows for non-linear further classification on different segments. The Layer B processing adds a priority to the hierarchy. Suitably, Layer B processing may be executed using segmentation, clustering k-means, K nearest neighbour, advanced classification, naive Bayes, random forests, logistic regression, or thresholding.

[00227] Layer C processing comprises principally context awareness or situational awareness. Layer C inherits the segmentation of Layer B and classification characteristics. This layer can understand its segmented surroundings and can further classify, predict for instance if an event is a threat, health, or risk disability. Suitably, Layer C processing may be executed using an Al transformer model, indexed parallel inference, CNN, or region-CNN (R-CNN). Table 1 below identifies for three different use case scenarios the processing that takes place at each layer. Example Scenario Layer A Layer B Layer C Security Object detection: person, vehicle, pet Advanced classification: child, adult, teen Determine threat: crowd, density, weapon Nutrition Object detection: person Segmentation: fruit, vegetable, meat Thresholding Feature Extraction Advanced classification Determine state of health Disability Object detection: person Advanced classification and segmentation: arm, leg, hand, etc. Determine risk of injury Table 1.

[00228] According to certain embodiments, for an IP camera as described above, the method further comprises receiving, at the edge computing device, the video data, and the indication of a detected object within the video data from a video camera associated with the building. However, while it is expected that the IP cameras with inbuilt object detection capability will be used, in some examples of the present invention cameras without such inbuilt capability may be used, in which case the edge computing device receives the video data from a video camera associated with the building and processes the received video data at the edge computing device to detect an object within the video data.

[00229] In some cases, the result of the Layer B processing at the edge computing device may comprise a forked decision. For instance, if a possible threat is identified, this may be termed priority 1 and result in the video data being passed to the fog computing device for further processing as described above. However, another outcome may be a lower priority (priority 2) such as the detection of a teenager carrying a football. This may not be deemed a threat and may result in appropriate action being taken without requiring fog processing.

[00230] In some examples, the edge computing device may be omitted, and the Layer B and Layer C processing executed by a fog computing device (either sequentially or in parallel). It will be appreciated that this loses the benefit of minimising the processing demand placed on the fog computing device (which may have other, unrelated roles).

[00231] The present invention may further comprise training a first machine learning algorithm using video data and indications of detected objects within the video data at the fog computing device; and providing the trained first machine learning algorithm to the edge computing device; wherein the first video data processing comprises processing the video data using the trained first machine learning algorithm. It will be appreciated that this use of the fog to train an Al algorithm or model for the edge to execute mirrors the split between model training and model execution described above in connection with monitoring or controlling a building system.

[00232] It will be appreciated that examples of the present invention can be realized in the form of hardware, software or a combination of hardware and software. Any such software may be stored in the form of volatile or non-volatile storage, for example a storage device like a ROM, whether erasable or rewritable or not, or in the form of memory, for example RAM, memory chips, device or integrated circuits or on an optically or magnetically readable medium, for example a CD, DVD, magnetic disk or magnetic tape or the like. It will be appreciated that the storage devices and storage media are examples of machine-readable storage that are suitable for storing a program or programs comprising instructions that, when executed, implement examples of the present invention.

[00233] Throughout this specification, the words “comprise” and “contain” and variations of them mean “including but not limited to”, and they are not intended to (and do not) exclude other components, integers, or steps. Throughout this specification, the singular encompasses the plural unless the context otherwise requires. In particular, where the indefinite article is used, the specification is to be understood as contemplating plurality as well as singularity, unless the context requires otherwise. Throughout this specification, the term “about” is used to provide flexibility to a range endpoint by providing that a given value may be “a little above” or “a little below” the endpoint. The degree of flexibility of this term can be dictated by the particular variable and can be determined based on experience and the associated description herein.

[00234] Features, integers, or characteristics described in conjunction with a particular aspect or example of the invention are to be understood to be applicable to any other aspect or example described herein unless incompatible therewith. All of the features disclosed in this specification, and / or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and / or steps are mutually exclusive. The invention is not restricted to the details of any foregoing examples. The invention extends to any novel feature or combination of features disclosed in this specification. It will be also be appreciated that, throughout this specification, language in the general form of “X for Y” (where Y is some action, activity or step and X is some means for carrying out that action, activity or step) encompasses means X adapted or arranged specifically, but not exclusively, to do Y.

[00235] Each feature disclosed in this specification may be replaced by alternative features serving the same, equivalent, or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.

[00236] The reader's attention is directed to all papers and documents which are filed concurrently with or previous to this specification in connection with this application and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference.

Claims

1. A computer implemented method of processing video data within a building, the method comprising:responsive to the detection of an object within video data, performing first video data processing upon the video data at an edge computing device associated with the building, wherein the first video data processing comprises further classification of the detected object or classification of a further object within the video data;determining, at the edge computing device, based on the first video data processing, whether second video data processing of the video data is required; andif it is determined that second video data processing is required, performing second video data processing at a fog computing device associated with the building;wherein the second video data processing determines whether to generate an output signal or a control signal for a building system based on the video data, the detected object, and the results of the first video data processing.

2. A method according to claim 1, wherein the first video data processing further comprises image segmentation.

3. A method according to claim 1 or claim 2, wherein the second video data processing further comprises context aware or situation aware processing of the video data.

4. A method according to any one of the preceding claims, further comprising: receiving, at the edge computing device, the video data, and the indication of a detected object within the video data from a video camera associated with the building.

5. A method according to any one of claims 1 to 3, further comprising:receiving, at the edge computing device, the video data from a video camera associated with the building; andprocessing the received video data at the edge computing device to detect an object within the video data.

6. A method according to any one of the preceding claims, wherein an object detected within the video data comprises one or more of:a person;a vehicle; oran animal.

7. A method according to claim 6, wherein if the detected object comprises a person, the first video data processing comprises further classification of the detected person as one of:a child;a teenager or young person;an adult; oran elderly person.

8. A method according to claim 6 or claim 7, wherein if the detected object comprises a person, the first video data processing comprises detecting and classifying one or more further object within the video data; andwherein the one or more further object comprises one or more of: a food item held by the detected person; ora potential weapon held by the detected person.

9. A method according to any one of claims 6 to 8, wherein if the detected object comprises a person, the first video data processing comprises identifying one or more body part visible in the video data from a group including:an arm;a leg; or a hand.

10. A method according to any one of the preceding claims, further comprising: training a first machine learning algorithm using video data and indications of detected objects within the video data at the fog computing device; andproviding the trained first machine learning algorithm to the edge computing device;wherein the first video data processing comprises processing the video data using the trained first machine learning algorithm.

11. A computer-readable storage medium having computer-readable program code stored therein that, in response to execution by a processor forming part of an edge computing device and a processor forming part of a fog computing device, cause the edge computing device and the fog computing device to perform the method of any one of the preceding claims.

12. A system for processing video data within a building, the system comprising:an edge computing device; anda fog computing device;wherein the edge computing device is configured to:responsive to the detection of an object within video data, perform first video data processing upon the video data, wherein the first video data processing comprises further classification of the detected object or classification of a further object within the video data; anddetermine, at the edge computing device, based on the first video data processing, whether second video data processing of the video data is required; andwherein the fog computing device is configured, if it is determined that second video data processing is required, to perform second video data processing;wherein the second video data processing determines whether to generate an output signal or a control signal fora building system based on the video data, the detected object, and the results of the first video data processing.

13. A system according to claim 12, further configured to execute the method of any one of claims 2 to 10.

14. A computer implemented method of operating an edge computing device to process video data within a building, the method comprising:responsive to the detection of an object within video data, performing first video data processing upon the video data, wherein the first video data processing comprises further classification of the detected object or classification of a further object within the video data; anddetermining, at the edge computing device, based on the first video data processing, whether second video data processing of the video data is required.

15. A method according to claim 14, wherein the edge computing device further performs part or the whole of the method of any one of claims 2 to 10.

16. A computer-readable storage medium having computer-readable program code stored therein that, in response to execution by a processor forming part of an edge computing device, cause the edge computing device to perform the method of claim 14 or claim 15.

17. An edge computing device comprising a processor and a memory storing executable instructions that, in response to execution by the processor, cause the edge computing device to perform the method of claim 14 or claim 15.

18. A computer implemented method of operating a fog computing device to process video data within a building, the method comprising:responsive to a determination, at an edge computing device, based on first video data processing, that second video data processing of the video data is required, performing second video data processing at a fog computing device associated with the building;wherein the second video data processing determines whether to generate an output signal or a control signal fora building system based on the video data, the detected object, and the results of the first video data processing.

19. A method according to claim 18, wherein the fog computing device further performs part or the whole of the method of any one of claims 2 to 10.

20. A computer-readable storage medium having computer-readable program code stored therein that, in response to execution by a processor forming part of a fog computing device, cause the fog computing device to perform the method of claim 18 or claim 19.

21. A fog computing device comprising a processor and a memory storing executable instructions that, in response to execution by the processor, cause the fog computing device to perform the method of claim 18 or claim 19.

Images