Local AI Inference

Abstract

Artificial intelligence (AI) is used to infer information from live video / audio feeds from doorbell cameras, security cameras, etc. Moving such capabilities from the cloud or edge can offer cost advantages, better privacy guarantees regarding monitored locations, and lower latency than other methods that use AI inference.

Description

[Technical field]

[0001] Artificial intelligence (AI) can be used to identify people, packages, or situations that require attention. [Background technology]

[0002] Analyzing video data, to the extent that facial recognition is involved, can involve a lot of hardware and software. Typically, such functionality is performed at scale across and on networks. However, given the ever-increasing capabilities of computing devices, there is a need to provide AI information processing at a local level. Summary of the Invention

[0003] Features, aspects, and advantages are described below with reference to the figures, which are intended to illustrate, but not limit, the invention, and in which like reference characters indicate corresponding features consistently throughout like embodiments. [Brief description of the drawings]

[0004] [Figure 1] FIG. 1 is a diagram of a doorbell system showing a doorbell / surveillance camera at the door of a house. [Diagram 2] A diagram depicting a usage scenario involving a doorbell / surveillance camera system. [Diagram 3] FIG. 2 is a call processing diagram relating to a live video call. [Figure 4] FIG. 2 illustrates a call processing flow associated with Pocket Inference. [Diagram 5] A diagram of a doorbell or security camera connected to a house in communication with a mobile computing device having an AI inference engine (e.g., a neural processor) and computer memory. [Figure 6]1 is a flowchart illustrating a process flow in which metadata (for AI inference, as determined by a neural processor on a mobile computing device) is stored on a mobile computing device. [Figure 7] FIG. 1 is a block diagram illustrating a mobile computing device having an AI inference engine with storage of AI inference metadata related to doorbell data stored locally at the mobile computing device. [Figure 8] 11 is a flowchart illustrating an example process flow in which metadata as determined by a neural processor using AI inference on a mobile computing device is stored remotely from the mobile computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Mobile computing devices such as smartphones contain artificial intelligence engines that can be used to exploit otherwise unused or underutilized computing power.

[0006] 1 illustrates a doorbell / surveillance camera system 100, showing a doorbell camera 102 at a door 104 of a house 106. An AI engine 108 is included in a smartphone 110.

[0007] 2 illustrates a diagram depicting a usage scenario involving the doorbell / surveillance camera system of FIG. 1. A delivery person 202 brings a package 204 to a house 106. A camera 102, which may be a doorbell camera or a surveillance camera, captures one or more images of the delivery person 202 approaching the house 106, with or without audio. Delivery data from the delivery person 202 is distributed to a network 210, as is the data from the camera 102. The network 210 can stream the delivery event data to the smartphone 110 of a user 214 directly or through a delivery information system 216.

[0008] Artificial intelligence can be used to identify people and objects associated with doorbell cameras and / or security cameras, and there is the cloud and edge (cloud refers to on-demand computer system resources, and edge refers to distributed data storage and information processing close to the source of the data) paradigm used in conjunction with the processing of video information.

[0009] In the case of streaming video, inference can be used to infer things from video and / or audio information using artificial intelligence (AI). AI inference can also be called computer vision or machine learning, and there are well-known ways to accomplish this. More specifically, a neural processor can use video and / or audio data from a doorbell camera and / or a security camera to infer certain things about that data. AI inference can tell if any face or body or package or pet is present in the recorded camera image or is identified in the recorded audio. Additionally, AI inference can be used in theft situations, such as when a package is stolen from a residence. The resulting inference information can be reported to the user through the cloud and on the user's mobile device. A relatively long time lapses between receiving a live call and receiving metadata about the live call. For example, inference can identify a person, package, vehicle type, etc.

[0010] In some cases, AI inferences can be performed by cloud computing. FIG. 3 illustrates a call processing diagram related to a live video call. Referring to FIG. 3, images or videos from a security camera or doorbell camera can be streamed during a live call to a mobile computing device, such as a smartphone, for decoding by a live stream decoder (not shown) and for display on a screen (not shown). This streamed information is typically then discarded, but a second feed of this information can be recorded in the cloud (denoted as back-end cloud). This information can be sent to an artificial intelligence service (cloud AI service), where processing in the cloud can make inferences from the detected data (about who, what, where in the stream, etc.). This inference information can be reported to the cloud (for storage in the back-end cloud) and then transferred to the mobile computing device, for example as metadata. More specifically, in the cloud, using tools such as “SageMaker™” (offered through Amazon Web Services™ (AWS)), there are ways at the edge to infer information from video by using libraries typically provided by chip vendors to perform analytics on video streams. This inference can answer questions about the video or image, such as is there a person there, is there a package there, is there a car there, etc.

[0011] Beyond the cloud and edge paradigm, there is fog. Fog refers to devices connected to the edge. Instead of performing inferences in the cloud or at the edge, inferences can be determined locally on the phone (e.g., in the fog, or at the "far" edge). Current smartphones typically include a neural processor, and this resource can be used advantageously in conjunction with artificial intelligence programs used to identify objects, events, etc. By using a neural processor that may already be present in the phone, it also obviates the need to place such capabilities at the doorbell location. In particular, newer phones are being sold with neural processors located near the regular general-purpose processor.

[0012] Additionally, processing power is more likely to be upgraded in a mobile phone than is upgrading doorbell processing hardware. Herein, AI inference in a mobile computing device is referred to as "in the pocket," where the pocket represents where a mobile phone might typically be. Making these inference decisions locally in a mobile computing device provides advantages. For example, a mobile phone often has much more processing power than is available at the edge, and using it is more likely to have lower latency than the cloud. Additionally, in-the-pocket (i.e., in the fog) processing is cheaper to use than processing in the cloud, since there is a charge each time functionality is used in the cloud. In contrast, a mobile phone offering similar functionality does not incur processing charges.

[0013] Additionally, a mobile phone may have real-time latency advantages because it may be closer to the device where AI inference is required. Thus, the mobile phone may be directly connected to such a device (e.g., a doorbell camera) over a local area network such as Wi-Fi rather than over the Internet. Even in cases where the mobile phone is very remote from the device and communicates over the Internet, in-the-pocket inference may not necessarily offer a latency advantage over cloud or edge inference, but the cost advantage remains.

[0014] An additional benefit of performing AI inference in the fog is that pocket inference provides privacy guarantees that do not require trust in the wording of service providers or other such entities. For example, cloud providers may claim that users can trust them not to reveal personal information, for example by showing videos to strangers who come to their door. They may further make claims such as "we will not see your information." With pocket inference, such trust is not required because the cloud provider will not allow users to see copies of surveillance images or videos, as the videos are not in the cloud.

[0015] FIG. 4 illustrates a call processing flow associated with Pocket Inference. FIG. 5 illustrates a diagram of a doorbell or security camera connected to a home in communication with a mobile computing device having an AI inference engine (e.g., neural processor) and computer memory. As shown in FIG. 4, video and / or audio data from the doorbell camera or security camera is streamed to a mobile computing device connected to the doorbell system. A decoder decodes this information and displays images on the mobile computing device. In addition, AI inference information from the streamed data is processed in conjunction with software running on the neural processor of the mobile computing device. The live stream decoding and AI inference may be simultaneous. In short, video frames from a live call are decoded. In addition to displaying the frames, the frames are processed by the neural processor in the background using inference before the frames are discarded. Metadata from the doorbell system (including the doorbell camera and / or other security cameras) may be stored in the cloud (denoted back-end cloud) (as shown in FIG. 4 and FIG. 5). Alternatively, the metadata may be stored locally on the mobile computing device.

[0016] FIG. 6 is a flowchart illustrating a process flow in which metadata (for AI inference, as determined by a neural processor on the mobile computing device) is stored on the mobile computing device.

[0017] FIG. 7 illustrates a block diagram showing a mobile computing device having an AI inference engine, where storage of AI inference metadata related to doorbell data is stored locally at the mobile computing device.

[0018] FIG. 8 is a flowchart illustrating a process flow in which metadata as determined by a neural processor using AI inference on a mobile computing device is stored remotely from the mobile computing device.

[0019] Distributed computing can also be used with AI inference performed on the mobile computing device. Thus, more than one device may be used to perform the necessary AI inference. For example, AI engine capacity may be borrowed from a mobile phone in a network associated with a blockchain detailing the occurrence of accounting, billing, crediting, etc. among participating smartphone owners. It is further contemplated that an older smartphone that does not have a neural processor can borrow the neural processing capacity of another smartphone. Furthermore, AI inference duty may be shared between a mobile device and, for example, a personal computer, tablet, etc. In other words, the AI ​​inference task may be offloaded to another device or a private server (or somewhere closer to the edge) under the control of the doorbell system owner. The application herein applies to real-time streaming video as well as voice. Many Internet of Things (IoT) applications involve sending telemetry data to the cloud where it is processed (with respect to decision making and inference execution). This may be done at the edge or in the fog. With regard to computer vision and analysis, many open source libraries are available that can be downloaded and used on mobile computing devices such as smartphones.

[0020] None of the steps described herein are required or essential. Any of the steps may be adjusted or modified. Other or additional steps may be used. Any portion of any of the steps, processes, structures, and / or devices disclosed or illustrated in one embodiment, flowchart, or example herein may be combined with, used with, or used in place of, any other portion of any of the steps, processes, structures, and / or devices disclosed or illustrated in another embodiment, flowchart, or example. The embodiments and examples provided herein are not intended as separate and distinct from one another.

[0021] The section headings and subheadings provided herein are non-limiting. The section headings and subheadings do not represent or limit the full scope of the embodiments described in the section to which they belong. For example, a section entitled "Topic 1" may include embodiments that do not belong to Topic 1, and embodiments described in other sections may be applied to and combined with embodiments described in the "Topic 1" section.

[0022] The various features and processes described above may be used independently or may be combined in various ways. It is intended that all possible combinations and subcombinations fall within the scope of the present disclosure. Furthermore, in some embodiments, certain methods, events, states, or process blocks may be omitted. Also, the methods, steps, and processes described herein are not limited to any particular order, and the blocks, steps, or states thereof may be performed in other suitable orders. For example, the tasks or events described may be performed in an order other than the order specifically disclosed. Multiple steps may be combined into a single block or state. The example tasks or events may be performed sequentially, in parallel, or in some other manner. Tasks or events may be added to or removed from the example embodiments disclosed. The example systems and components described herein may be configured differently than as described. For example, elements may be added to, removed from, or rearranged compared to the example embodiments disclosed.

[0023] Conditional language used herein, particularly "can," "may," "may," "might," "for example," and the like, is generally intended to mean that certain embodiments include certain features, elements, and / or steps while other embodiments do not, unless expressly stated otherwise or understood otherwise in the context in which it is used. Thus, such conditional language is generally not intended to imply that features, elements, and / or steps are somehow required for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included or performed in any particular embodiment, with or without author input or prompting. Terms such as "comprises," "includes," "having," and the like, are synonymous and are used inclusively in an open-ended manner and do not exclude additional elements, features, acts, operations, etc. Additionally, the term "or" is used in an inclusive (not exclusive) sense, e.g., when used to connect a list of elements, the term "or" may mean one, some, or all of the elements in the list. Collocations such as "at least one of X, Y, and Z," unless expressly stated otherwise, are understood differently in the context in which they are used as generally meaning that an item, term, etc. may be either X, Y, or Z. Thus, such collocations are not generally intended to suggest that a particular embodiment requires that at least one X, at least one Y, and at least one Z, respectively, be present.

[0024] The term "and / or" means that some embodiments apply with "and" and some apply with "or". Thus, A, B, and / or C may be substituted for A, B, and C when written in one sentence, and may be substituted for A, B, or C when written in another sentence. A, B, and / or C means that some embodiments may include A and B, some embodiments may include A and C, some embodiments may include B and C, some embodiments may include only A, some embodiments may include only B, some embodiments may include only C, and some embodiments may include A, B, and C. The term "and / or" is used to avoid unnecessary redundancy.

[0025] Although certain exemplary embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the invention disclosed herein. Thus, nothing in the above description is intended to imply that any particular feature, characteristic, step, module, or block is essential or essential. Rather, the novel methods and systems described herein may be embodied in a variety of other forms, and further, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the spirit of the invention disclosed herein.

Claims

1. receiving, through a mobile computing device, a feed selected from the group consisting of a video feed, an audio feed, and combinations thereof, wherein the video feed and the audio feed occur in association with a live communication session; analyzing the video feed using an artificial intelligence (AI) engine on the mobile computing device to identify one or more parameters associated with the video feed; storing metadata associated with the one or more parameters; A method comprising:

2. The method of claim 1 , wherein the metadata associated with the one or more parameters is stored on the mobile computing device.

3. The method of claim 1 , wherein the metadata associated with the one or more parameters is stored at a location remote from the mobile computing device.

4. The method of claim 1 , wherein the AI ​​engine is configured to identify the one or more parameters using AI reasoning.

5. The method of claim 4 , wherein the one or more parameters are indicative of a person.

6. The method of claim 4 , wherein the one or more parameters are indicative of movement.

7. The method of claim 4 , wherein the one or more parameters indicate a presence of a visitor.

8. The method of claim 4 , wherein the one or more parameters are indicative of facial recognition data.

9. The method of claim 4 , wherein the one or more parameters are indicative of one or more audible sounds.

10. The method of claim 4 , wherein the one or more parameters are indicative of a package delivery or a package theft.

11. A video camera and a computer readable non-transitory programmable article comprising code executable by a processor in a mobile computing device to cause the processor to identify one or more parameters associated with a video feed from the video camera; a memory configured to receive metadata associated with the one or more parameters; A system comprising:

12. 12. The system of claim 11, wherein the one or more parameters are selected from the group consisting of people, movement, visitor presence, facial recognition data, one or more audible sounds, and package delivery.

13. The system of claim 11 , wherein the memory is located within the mobile computing device.

14. The system of claim 11 , wherein the memory is part of a cloud storage located remotely from the mobile computing device.

15. 12. The system of claim 11, wherein the processor is configured to use AI inference to identify the one or more parameters associated with the video feed.

16. The system of claim 11 , wherein the video camera is a doorbell camera.

17. The system of claim 11 , wherein the video camera is a stand-alone camera.

18. The system of claim 11 , wherein the video feed is a live video feed.

19. 1. A computer-readable, non-transitory, programmable product comprising: code executable by a processor to cause a processor to analyze a video feed using an artificial intelligence (AI) engine on a mobile computing device to identify one or more parameters associated with the video feed, the code further causing the processor to receive the video feed from a video camera located remotely from the mobile computing device.

20. 20. The computer readable non-transitory programmable product of claim 19, wherein the code further causes the processor to identify metadata associated with the one or more parameters.

21. 20. The computer readable non-transitory programmable product of claim 19, wherein the one or more parameters are indicative of a person.

Images