Cascaded feature detectors for stream processing efficiency

Abstract

A computer-implemented method of data processing includes receiving an input data stream; identifying one or more first content features of the data stream by a first algorithm; determining one or more portions of the data stream to be stored in a first buffer based on at least one output of the first algorithm; storing the determined portions in the first buffer; identifying one or more second content features of the data stream using the stored portions of the data stream by a second algorithm; and generating output based on the identified second content features for output to a user, wherein the second algorithm is more computationally complex than the first algorithm.

Description

[Technical Field]

[0001] The present invention relates to the analysis of streams of data where a large portion of the data stream is not "interesting", and in particular the present invention relates to the analysis of data from, for example, video or security cameras, microphones for voice control, or streams of data for medical diagnostics. [Background technology]

[0002] There are many applications in which electronic systems seek to monitor streams of data over time, but often there is not much "interesting" happening.

[0003] For example, there are "intelligent video cameras" used in security, process monitoring, healthcare, and many other applications, where the data stream consists of video image frames and may include an audio stream. The present disclosure applies equally to any other application involving monitoring a stream of data, such as monitoring sensor output from an industrial process, listening to voice commands for a smart speaker, or processing a set of data previously written to a storage medium (e.g., the automated processing of a video file or a text file representing a work of literature).

[0004] Existing implementations sometimes detect changes in data streams (e.g., motion detectors in video) during periods when little "interesting" is occurring in order to minimize the required processing, and therefore computational hardware, and therefore cost and power. However, understanding the intricacies of what is "interesting" remains a challenge. Because it is generally beneficial to minimize the chance of "false negatives," this leads to high "false positive" rates, resulting in higher than optimal processing costs and associated hardware costs.

[0005] Using the example of an intelligent video camera, a processing algorithm (e.g., a neural network) can be used to fully process an image or a recent stream of images. This requires significant computational power. Figure 1 is a schematic diagram of such a basic intelligent video camera 1, comprising a video camera 110, a neural network 120, and an output 130. The computer hardware always runs the neural network for every frame (i.e., 100% hardware cost and power consumption).

[0006] If the stream of images changes very little, existing implementations may include a motion detector to determine whether an image is substantially similar to the previous image before sending it to the main algorithm. Figure 2 shows an intelligent video camera 2 with such motion detection, comprising a video camera 210, a motion detector 220, a neural network 230, and an output 240. If there is continuous motion in the scene over a period of time, the main algorithm 220 must process every video frame. Therefore, the computing hardware must match the capabilities of the continuously running algorithm.

[0007] Any savings from running a motion detector are only in power, not hardware cost; the computer hardware still needs to be able to run the neural network 220 on every frame, so the hardware cost is 100%, but the power consumption on average is reduced to, say, 10%.

[0008] In some conventional implementations, the video camera can offload more complex algorithms to a server on the network, meaning the camera hardware is relatively inexpensive, but the network connection must be able to handle the peak data rates from the camera, which also comes at a significant cost.

[0009] The present invention is directed to reducing the processing required both during periods when little "interesting" stuff is happening, and during long periods of peak operation, reducing the power requirements and cost of the system's computing hardware. Summary of the Invention

[0010] This disclosure describes techniques, methods, systems, and other mechanisms for data analysis and image processing.

[0011] According to a first aspect of the present invention, there is provided a computer-implemented method comprising: receiving an input data stream; identifying one or more first content features of the data stream by a first algorithm; determining one or more portions of the data stream to store in a first buffer based on at least one output of the first algorithm; storing the determined portions in the first buffer; identifying one or more second content features of the data stream by a second algorithm using the stored portions of the data stream; and generating output for output to a user based on the identified second content features, wherein the second algorithm is more computationally complex than the first algorithm.

[0012] The method may include determining one or more stored portions of the data stream to delete from the first buffer based on at least one output of the first algorithm, and deleting the determined portions from the first buffer. The first algorithm may include multiple layers of processing, and the portion of the data stream that is stored is determined based on the higher layer output of the first algorithm. The characteristics of the first content may be substantially the same as the characteristics of the second content. The first algorithm and the second algorithm may be trained using a common training data set and a common training objective.

[0013] The method may include pre-processing one or more portions of the data stream to be stored in the first buffer before storing the determined portions in the first buffer. The determining may include determining a substantially fixed proportion of the data stream to be stored in the first buffer.

[0014] The method may include updating the first algorithm based on identification of the one or more second content characteristics by the second algorithm. One or more of the characteristics of the first content identified by the first algorithm may be stored in a second buffer.

[0015] The determination to store one or more portions of the data stream in the first buffer may be based at least in part on characteristics of the first content stored in the second buffer.

[0016] The data stream may be a video stream including a plurality of image frames, and the first content feature and the second content feature may be image features of the image frames. The image features may correspond to people and the generated output may be based on an estimated number of people in the video stream. The image features may correspond to one or more diagnostic indicators, and the generated output may include at least one diagnostic indicator to aid in diagnosis.

[0017] According to a second aspect of the present invention there is provided a data processing apparatus comprising an input for receiving an input data stream, a processor and an output, the apparatus being configured to perform the method of the first aspect.

[0018] According to a third aspect of the present invention, a storage medium is provided and configured to store instructions which, when executed by a processor, cause the processor to perform the method of the first aspect. Optional features are as set out in the dependent claims. [Brief explanation of the drawings]

[0019] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. [Figure 1] 1 is a schematic diagram of a basic intelligent video camera. [Figure 2] 1 is a schematic diagram of a video camera with motion detection. [Figure 3] 1 is a schematic diagram of a video camera with motion detection and dynamic threshold buffering, according to one embodiment. [Figure 4] FIG. 1 is a schematic diagram of a video camera with a cascade of algorithms and caches according to one embodiment. [Figure 5] FIG. 1 is a schematic diagram of a deep convolutional neural network according to one embodiment. [Figure 6] 1 is a schematic diagram of a video camera with motion detection and a cascade of algorithms and caches, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] 3 illustrates an image capture and image processing apparatus 3 with motion detection and dynamic threshold buffering, according to one embodiment. The apparatus 3 comprises a video camera 310, a motion detector 320, a dynamic threshold buffer 330, a neural network 340, and an output 350.

[0021] The video camera 310 is configured to receive an input data stream comprising video data. The video camera 310 is configured to capture the video data in the form of a plurality of video image frames. The captured plurality of video image frames may form the input data stream. The motion detector 320 is configured to receive the plurality of video image frames from the video camera 310. The motion detector 320 is configured to detect motion in one or more of the video image frames. The motion detector 320 may perform an image processing algorithm to detect motion. For example, the motion detector 320 may be configured to detect changes in pixel intensity in the input data stream. The motion detector 320 may be thought of as identifying changing pixels corresponding to one or more content features of the input data stream, such as moving objects. For example, the motion detector 320 may identify the presence of one or more moving objects by detecting motion in the video data. The motion detector 320 may be configured to cache the plurality of video image frames in which motion is detected, for example, by writing them to disk.

[0022] The dynamic threshold buffer 330 is configured to store one or more portions of the input data stream. The dynamic threshold buffer 330 is configured to store one or more of the plurality of video image frames in which motion is detected. The dynamic threshold buffer 330 is configured to determine the portions of the input data stream to store based on the motion detected by the motion detector 320. For example, the dynamic threshold buffer 330 may reject portions of video that are below a dynamic motion threshold.

[0023] The dynamic threshold buffer 330 may be configured to determine a substantially fixed percentage of the data stream to store in the first buffer. For example, the dynamic threshold buffer 330 may be fixed at 90% rejection, such that 90% of the video image frames are discarded and 10% are retained. The dynamic threshold buffer 330 may use dynamic thresholding techniques to determine the sensitivity of the dynamic motion threshold to movement. For example, during periods of relatively low movement in the scene, the dynamic threshold may be lowered until a consistent 10% of the images are cached, even at this reduced motion level, allowing the neural network 340 to process the reduced data stream.

[0024] During periods of increasing motion, the buffer may reject portions of the video that fall below a dynamically increasing threshold. The dynamic threshold buffer 330 may determine one or more stored portions of the data stream to remove from the first buffer. For example, when a person walks in front of the camera, all incoming frames will exhibit much higher motion estimates than the frames currently in the buffer, evicting many low-motion frames from the buffer. This allows the buffer to effectively buffer short periods of interesting activity.

[0025] As motion continues in the scene, the dynamic threshold automatically increases, and eventually the system stabilizes at the new threshold, leaving only 10% of the images stored in the cache. In this way, subsequent neural networks 340 can run on a constant percentage of the total input data stream. The computer hardware only needs to have the capacity to run the neural network 340 on 10% of the data. The neural network 340 can only process the highest priority images in the cache at a time (priority is typically a combination of age and motion level), and the dynamic threshold adjustment algorithm ensures that the cache always contains enough data for the algorithm to continue processing.

[0026] In this way, device 3 can reduce the amount of computation required to evaluate certain portions of the input data stream as "uninteresting" and discard them. This provides a principled way of prioritizing "important" data (and thus reducing latency). By allowing neural network 340 to process only a fixed percentage of the most "interesting" portions of the data, it is possible to save on both power and computational hardware costs.

[0027] In this way, power consumption can be reduced by approximately 10%, and hardware costs can be reduced by approximately 10%. In other embodiments, the retention threshold may be set, for example, at 5% or 15%, depending on system requirements. The retention threshold may be anywhere within the range of 0% to 50%.

[0028] In some examples, the retention threshold may be adjusted to allow dynamic improvement of the system's false positive / false negative rate based on the data stream being observed. For example, the threshold may be adjusted for a particular intelligent video camera to improve the system based on the scene being observed by that camera.

[0029] The neural network 340 is configured to identify one or more content features of the input data stream. The neural network 340 is an example of an algorithm for identifying content features. The neural network 340 is configured to use stored portions of the data stream. It can be seen that the neural network 340 is more computationally complex than the motion detector 320.

[0030] In some examples, the input data stream may be a video stream comprising multiple image frames, and the content features may be image features of the video image frames. The content features may correspond to people, and the neural network 340 may be configured to estimate the number of people in the video stream. The output of the final layer of the neural network 340 may be a single value that is the number of people in the video stream.

[0031] Output 350 is configured to generate an output based on the identified content features provided by neural network 340 for output to a user.

[0032] 4 shows an image processing device 4 with a cascade of algorithms with a cache, according to one embodiment. In the above-described embodiment, there may be differences in the definition of "interesting" between the motion detector 320 and the neural network 340. An ideal motion detector would be sensitive only to elements of the image to which the neural network is sensitive, which essentially means that it would produce the same output as the neural network itself. This means that it would require similar computational costs and would not offer any savings.

[0033] In some embodiments, a cascade of algorithms of increasing computational complexity may be trained. The algorithms may be neural networks. Each neural network may be trained on similar training data. The cascade of algorithms of increasing computational complexity may each attempt to more closely detect "interesting" features than the previous one. In some embodiments, earlier stages of the cascade may be dynamically improved over time to more closely match the output of later stages of the cascade given observed actual data.

[0034] The device 4 comprises a video camera 410, a first neural network 420, a buffer 430, a second neural network 440, and an output 450. Elements not described are substantially as described with respect to FIG.

[0035] The video camera 410 is configured to receive an input data stream having video data, and the video camera 410 is configured to immediately pass the input data stream to a first neural network 420.

[0036] The first neural network 420 is configured to identify one or more content features of the input data stream. The neural network 420 is an example of an algorithm for identifying content features. The first neural network 420 may be thought of as acting as a coarse filter on the image of interest.

[0037] In some examples, the input data stream may be a video stream including multiple image frames, and the content features may be image features of the video image frames. The image features may correspond to people, and the neural network 420 may be configured to estimate the number of people in the video stream. The output of the final layer of the neural network 420 may be a single value that is the number of people in the video stream.

[0038] Neural network 420 further includes one or more additional layers within the network. The additional layers may respond to different elements within an image. For example, one or more layers may be configured to respond to human-shaped blocks, different contrasts, etc. In one or more layers closer to the network's output, due to the hierarchical structure of deep convolutional neural networks, there may be an internal representation of the image that is more responsive to desired content features. These layers are sometimes referred to as "higher layers." For example, one or more higher layers may respond more strongly to the number of people in a room than to other content features of the dataset used for training.

[0039] FIG. 5 shows a diagram of a neural network 420 according to one embodiment. The neural network 420 includes multiple processing layers: an input layer 421, one or more hidden layers 422, an upper layer 423, one or more output layers 424, and an output 425.

[0040] The upper layer 423 before the output layer 424 is highlighted, and the activation of the upper layer 423 is shown as a vector. In some examples, the portion of the input data stream to be stored in the buffer may be determined based on the upper layer output.

[0041] These "higher layer" activations can be treated as points in a high-dimensional space that summarize the image processed by the neural network. In this way, the distance between any two points can provide a measure of the "similarity" between the two images. In some instances, this notion of "similarity" may specifically refer to a change in the algorithm's output, such as the number of people in a room.

[0042] Returning to Figure 4, buffer 430 is configured to store one or more portions of the input data stream. Buffer 430 may be configured to store one or more of the content features identified by neural network 420. Buffer 430 may also store one or more images of interest from the input data stream prior to ingestion by second neural network 440. Apparatus 4 places buffers between algorithms so that high priority changes in the input data stream can be propagated quickly through the cascade.

[0043] In some examples, the method may include preprocessing one or more portions of the input data stream to be stored in buffer 430 before storing the determined portions in buffer 430. In some embodiments, adding / discarding to / from buffer 430 may be controlled at least in part by a mathematical distance metric. The mathematical distance metric may be measured between the current sample under consideration and either samples collected around it in time or a portion of samples in a cache. For example, an item is likely to enter buffer 430 if the distance metric is large compared to samples adjacent in time. The distance metric may indicate that the data stream has changed in a way that is deemed interesting by neural network 420.

[0044] In some examples, determining which portion or portions of the input data stream to store in buffer 430 may be based at least in part on features of one or more content stored in a second buffer. For example, the second buffer may be configured to store "upper layer" outputs from neural network 420. The second buffer may continuously evaluate the mathematical distance between all of the stored images to maximize the diversity of the cached content. In this way, buffer 430 can be prevented from filling up when an image toggles between two interesting states. For example, if a person repeatedly enters and exits the frame, the person count may change frequently over time, but in a way that is not considered particularly significant.

[0045] The choice of distance metric (e.g., Euclidean distance, cosine distance), preprocessing of point dimensions (e.g., normalization, combining), and selection of depth from the output of the neural network 420 are engineering choices. The distance metric may be used by a buffer 430 between the first neural network 420 and the second neural network 440 to determine which images are interesting enough to retain. For example, if the output of the neural network 420 contains more features, the distance metric may incorporate this information. For example, if the neural network 420 provides an estimate of the gender ratio of people in a room, a scene containing 10 men may be considered only somewhat similar to a scene containing 10 women.

[0046] The second neural network 440 is configured to identify one or more content features of the input data stream. The neural network 440 is an example of an algorithm for identifying content features. The neural network 440 is configured to use a stored portion of the data stream in the buffer 430. The neural network 440 may be more computationally complex than the neural network 420. The second content features identified by the second neural network 440 may be substantially the same as the first content features identified by the first neural network 420. The first neural network 420 and the second neural network 440 may be trained using a common training data set and a common training objective.

[0047] In some examples, the input data stream may be a video stream including multiple image frames, and the second content feature may be an image feature of the video image frame. The image feature may correspond to people, and the second neural network 440 may be configured to estimate the number of people in the video stream. The output of the neural network 440 may be a single value that is the number of people in the video stream.

[0048] Output 450 is configured to generate an output based on the identified content features provided by second neural network 440 for output to a user.

[0049] By implementing device 4, the "false positive" and "false negative" rates of the algorithmic cascade can be reduced compared to systems with simple digital signal processing steps followed by complex algorithms, such as device 3. In this way, both the peak and average computational power can be further reduced compared to existing approaches, and therefore the associated hardware costs can also be reduced.

[0050] Such an application of device 4 can utilize a lower complexity algorithm as a filter for only some content features that a more complex algorithm is configured to identify in the data stream. In this way, it is possible to optimize complexity, hardware cost, or power consumption for those hardware elements that can be efficiently used to select portions of the data stream to pass to the more complex algorithm. In some examples, a face recognition algorithm is relatively complex because it is configured to extract a large number of features from an input image. However, a lower complexity algorithm may identify a subset of face-related features (e.g., the presence / absence of a face and landmarks such as the nose, eyes, and ears).

[0051] Apparatus 4 can provide a disciplined approach to prioritizing the “diversity” of samples in a dataset (i.e., of samples in a cache or sent to end-stage processing). Diversity can be defined as a mathematical distance function between samples. This function can define what content features in the samples are most noteworthy. In some examples, for example, in an intelligent video camera for crowd surveillance, this capability can enable automatic prioritization of face detection across different faces or different views of the same face. In some examples, this functionality can be achieved without requiring computationally intensive and expensive face detection algorithms to be run on every sample.

[0052] Prioritizing buffer diversity using a lower complexity algorithm may result in the more complex algorithm being run on a wider range of facial positions / orientations within the data stream. This can be achieved without prioritizing the uniqueness of the faces themselves. In some instances, some or all of the content features identified by a lower complexity algorithm may differ from the content features identified by a more complex algorithm. A lower complexity algorithm can effectively act as a filter on the input data stream to identify portions of the input data stream that are of more interest to the more complex algorithm. For example, a lower complexity algorithm may be configured to identify a person's head / neck / shoulders in order to crop around each portion of the image that is likely to contain a face, while a more complex algorithm may be configured to perform facial recognition on the cropped partial images.

[0053] In some embodiments, the concept of diversity may be enforced across multiple cache layers or even the entire system. For example, the vectors of the upper layer outputs described in FIG. 5 may be used so that each new image is compared not only with the contents of the cache immediately following the initial neural network, but also with the contents of one or more subsequent caches. In some examples, images that are highly similar to images that have already passed through the entire filter chain may be discarded. In some examples, vectors associated with images that have passed through or been discarded by subsequent stages may be kept in a circular buffer. In this way, images that appear interesting to a simpler front-end network but are determined to be uninteresting by subsequent stages are not unnecessarily reprocessed if a very similar image is seen again. This approach may ensure that storage space for images processed by subsequent stages can be prioritized over images that are different from those already seen.

[0054] In some embodiments, incorporating image age into cache acceptance / rejection logic can help ensure image freshness along with diversity. In some examples, the distance metric between two images may be divided by the difference in age. In this way, the similarity threshold for cache inclusion may be configured to be age-dependent.

[0055] The "velocity" of an image through a cache may be defined as the amount of time the image spends in the cache before being evicted. In some examples, an image may be made available to the second stage of the cascade once it has spent longer than a predetermined (and possibly dynamically adjustable) period in the first-level cache. In this way, a maximum average throughput of images may be enforced between stages of the cascade. In this way, cases can be avoided where a promising image is rapidly replaced by an even more promising image with a similar distance metric.

[0056] For example, in the case of a person walking and stopping, two images may be semantically similar (i.e., a low distance metric), but the acceptance logic may prioritize the frame with less inter-frame motion (and therefore blurrier). It would be inefficient to immediately pass a potentially blurry image to the second stage for processing without ensuring that at least a small amount of time has passed without a better image being captured. The concept of speed also prevents the cold-start problem, where the cache is initially emptied entirely. In such a case, virtually all of the first images observed may end up in the cache, and most of these images may be evicted soon thereafter in favor of slightly better or more diverse images. By enforcing a minimum time spent in the cache before moving on to the next level, the system can avoid filling the next level's network and cache with redundant data. This can be particularly advantageous when the media for different caches have different write speeds (e.g., RAM vs. SSD).

[0057] In some embodiments, one or more intermediate algorithms may output values ​​in a meaningful format (e.g., an estimate of the number of people, as opposed to simply a dimensionless number(s)). In some examples, the output values ​​may be used to prioritize or deprioritize portions of a dataset based on these previous outputs. For example, in a people-counting video camera used for crowd surveillance, one or more images may be discarded if the count is estimated below a certain level by a very small neural network. When the output from an algorithm is relatively complex, different elements of the output may be more useful at different stages of the cascade. For example, the location of an object of interest may be as useful as the presence or absence of that object. In some examples, an image may be deprioritized if the object of interest is likely not present, even if it has a large similarity distance from the previous image.

[0058] In some embodiments, it is possible to train a filter with multiple outputs. Some outputs may be used as a quality calculation to determine whether to keep or discard an image. Some outputs may be used as a diversity calculation to minimize repeatedly processed images. In the case of a face recognition system, the quality output may be a classifier indicating the probability of a face in the image, and the diversity output may be an estimate of the angle of the face to the camera. In some examples, the higher-layer component encoding the quality output may be separate from the component encoding the diversity output. In this way, using a vector of higher-layer activations for the distance metric when selecting images to put into the cache does not inadvertently maximize the quality diversity in the cache. In this way, the diversity filtering step can avoid generating the widest possible range of possible faces appearing in the image, rather than the widest possible range of face angles. For example, the higher-layer activations that determine quality may represent the likelihood of a face being present in different locations in the image. Including these components in the distance metric for diversity may inadvertently prioritize the same face at the same angle relative to the camera as you move around the image. In some examples, the set of features on which diversity is calculated may be less than the full set of features generated by the network.

[0059] 6 illustrates an image processing device 6 having a cascade of algorithms and caches, according to one embodiment. In some examples, an intelligent video camera may include device 6 and be configured for a specific application, such as counting the number of people in a room. Since the output of the main algorithm for the number of people in the room will not change if there is no motion between frames, device 6 may include a simple motion detector as the first element of the cascade.

[0060] The device 6 may be implemented with a simple motion detector at the front end. Subsequent elements in the cascade may be deep convolutional neural networks of varying complexity. The device 6 may include a series of deep convolutional neural networks (CNNs), such as CNNs with a modified "EfficientNet" architecture. Some architectures in the EfficientNet family are known as BO, B4, and B7. The neural networks may be configured to operate on copies of images downsampled to various degrees. All neural networks may be trained for the same task, such as counting people. While the system is described based on neural networks, the invention is applicable to any alternative digital signal processing technology.

[0061] The device 6 comprises a video camera 610 , a motion detector 620 , a first neural network 630 , a first cache 640 , a second neural network 650 , a second cache 660 , a third neural network 670 and an output 680 .

[0062] The video camera 610 is configured to receive an input data stream having video data. The video camera 610 is configured to capture the video data in the form of a plurality of video image frames. The captured plurality of video image frames may form the input data stream.

[0063] The motion detector 620 is configured to receive a plurality of video image frames from the video camera 610. The motion detector 620 is configured to detect motion in one or more of the video image frames. The motion detector 620 may perform an image processing algorithm to detect motion. For example, the motion detector 620 may be configured to detect changes in pixel intensities in the input data stream. The motion detector 620 may be thought of as identifying one or more content features of the input data stream, such as changing pixels that may correspond to moving objects. For example, the motion detector 620 may identify the presence of one or more moving objects by detecting motion in the video data. The motion detector 620 may be configured to provide the plurality of video image frames in which motion is detected to the first neural network 630.

[0064] The first neural network 630 is configured to identify one or more content features of the input data stream. The neural network 630 is an example of an algorithm for identifying content features. The first neural network 630 may be a B0 neural network. The B0 neural network may have a typical computational requirement for an image of 0.4 GFLOPs and a top-1 ImageNet error of 23.7%. The first neural network 630 may be thought of as acting as a coarse filter for the image of interest.

[0065] In some examples, the input data stream may be a video stream including multiple image frames, and the content features may be image features of the video image frames. The image features may correspond to people, and the neural network 630 may be configured to estimate the number of people in the video stream. The output of the final layer of the neural network 630 may be a single value that is the number of people in the video stream.

[0066] The first cache 640 is configured to store one or more portions of the input data stream. The first cache 640 may be configured to store one or more content features identified by the first neural network 630. The first cache 640 may store one or more images of interest from the input data stream prior to ingestion by the second neural network 650. The apparatus 6 is arranged with buffers between algorithms to ensure that high priority changes in the input data stream are propagated quickly through the cascade.

[0067] The second neural network 650 is configured to identify one or more content features of the input data stream. The second neural network 650 is an example of an algorithm for identifying content features. The second neural network 650 may be a B4 neural network. The B4 neural network may have a typical computational demand for an image of 4.2 GFLOPs and a top-1 ImageNet error of 17.4%.

[0068] Given the difference in computational complexity between the B0 and B4 neural networks, it may be desirable for the first cache 640 between them to hold a maximum of approximately 0.4 / 4.2 of the frames between these stages. In this way, computation is roughly distributed across the network. In some examples, the first cache 640 may be configured to hold approximately 10% of the frames.

[0069] The second cache 660 is configured to store one or more portions of the input data stream. The second cache 660 may be configured to store one or more content features identified by the second neural network 650. The second cache 660 may store one or more images of interest from the input data stream prior to ingestion by the third neural network 670.

[0070] The third neural network 670 is configured to identify one or more content features of the input data stream. The third neural network 670 is an example of an algorithm for identifying content features. The third neural network 670 may be a B7 neural network. The B7 neural network may have a typical computational demand for an image of 37 GFLOPs and a top-1 ImageNet error of 15.6%.

[0071] Considering the difference in computational complexity between the B4 neural network and the B7 neural network, the second cache 660 between them preferably holds a maximum of approximately 4.2 / 37 of the frames between these stages. In this way, computation is roughly distributed across the network. In some examples, the second cache 660 may be configured to hold approximately 10% of the frames.

[0072] Output 680 is configured to generate an output based on the identified content features provided by third neural network 670 for output to a user.

[0073] In device 6, the hardware must be capable of processing approximately 1.2 GFLOPs per input image. However, output 680 provides images processed by a final neural network with 37 GFLOPs. In this way, device 6 can provide approximately a 30-fold reduction in required computational power. A motion detector can reduce average power consumption, but not peak power consumption or hardware costs. The cascaded network in device 6 is configured to generate a final output for only 1% of images on average, which is acceptable when considering, for example, a security camera that spends most of its time viewing relatively static scenes.

[0074] In some embodiments, device 6 may enable early stopping if lower-level networks produce results with a certain degree of confidence. In some examples, the output of the intermediate neural network is used to assist in prioritization in the cache, thereby enabling early stopping. In some examples, the intermediate neural network may output an indication of confidence. For example, an object classifier can provide a probability of match so that clear matches are revealed. This may allow the system to use the intermediate-level network output instead of running the final-stage network when data is easily classified. This approach may save more power or provide higher throughput.

[0075] Furthermore, if multiple stages in a cascade are attempting to provide the same output, it is possible to improve the accuracy of earlier stages based on the output of later stages for the same image.

[0076] In some embodiments, one or more front-end neural networks can be updated based on the identification of one or more content features by the back-end neural network. For example, one or more of the first neural network 630 and the second neural network 650 may be updated based on the output of the third neural network 670.

[0077] In some examples, images where the output of a previous stage does not match the output of a subsequent stage may be used to improve the reliability of the previous stage. A backpropagation algorithm may be used to calculate weight updates for the previous neural network by using the output of the subsequent neural network as ground truth for loss function evaluation. In some examples, a history of upper layer outputs for all images contributing to such training may be maintained. New training data may be selected based on both the output error and its diversity compared to the history of previous training data. In this way, a maximally useful set of updates to the neural network weights can be calculated. Furthermore, these weight updates are inherently privacy-preserving because the actual outputs are not encoded. Therefore, it is safe to send them to a central server to update the performance of many devices connected to such a network.

[0078] In some embodiments, one or more top layers, or output layers, are kept the same for multiple neural networks. In this way, the outputs of the upper layers immediately below the top layer can be directly compared between different neural networks. This has two advantages. First, if diversity is desired in the input data stream, it is possible to maximize the diversity of the cache stage between all neural networks without having to individually store the upper layer outputs from all neural networks. Second, training from the upper layers (backpropagation) can provide more information for training the network than training from a more restricted output set. This can result in better training from a given number of training images.

[0079] In some implementations, the image features may correspond to one or more diagnostic indicators, and the generated output may include at least one diagnostic indicator to aid in diagnosis.

[0080] In some implementations, the input data stream may be an audio stream. For example, the input data stream may be audio input from a microphone, such as for hot word detection by a voice-activated assistant. The voice assistant device spends most of its time listening for the hot word before sending a buffer of captured audio for complete analysis of the command. In some examples, the hot word may be a verbal phrase, such as "OK assistant." The hot word must be detected almost instantly, but with the lowest possible average computational cost due to the large amount of silence or listening time for something other than the hot word. In some implementations, existing digital signal processing algorithms may be incorporated to achieve acceptable false positive / false negative rates. For example, an algorithm may be incorporated to detect a threshold level of sound or to perform a Fourier transform of audio to check the frequency range of the human voice. As mentioned above, subsequent stages may be implemented in cascade to achieve better performance or lower computational / power requirements.

[0081] The hot word detection problem described above is an example of triggering when specific content is observed in the input data stream. This is related to semantic motion detection, where the distance of a current sample of data is calculated relative to a database of other samples. In some instances, this may trigger the capture of hot words, etc., where the database is a variety of trigger words. In some instances, the capture of a lack of meaningful motion, etc., may be avoided if the database is a history of recent samples. In some instances, the use of sample digests, e.g., vectors of upper layer activations, allows for dense storage of such databases. The more semantically related the digests, the denser the database can be.

[0082] Some or all of the different aspects of the present invention (variety, quality, movement, age, speed, content) can be combined as needed for a particular application. Any portion of the present invention may be split between processing devices, for example, by placing network connections between buffering components.

[0083] Although aspects of the invention herein have been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present invention. It is therefore to be understood that numerous modifications can be made to the illustrative embodiments and other arrangements can be devised without departing from the scope of the invention as defined by the appended claims.

Claims

1. 1. A computer-implemented method of data processing, comprising: receiving an input data stream; identifying, by a first algorithm, one or more first content characteristics of the data stream; determining one or more portions of the data stream to be stored in a first buffer based on at least one output of the first algorithm; storing the determined portion in a first buffer; identifying, by a second algorithm, one or more second content characteristics of the data stream using the stored portion of the data stream; generating an output for output to a user based on the identified second content characteristics; The second algorithm is more computationally complex than the first, A data processing method, wherein the characteristics of the first content are substantially the same as the characteristics of the second content.

2. The method of claim 1 , wherein the second algorithm analyzes only the stored portion of the data stream.

3. determining one or more stored portions of the data stream to be deleted from the first buffer based on at least one output of the first algorithm; removing the determined portion from the first buffer; 3. The method of claim 1 or 2, further comprising:

4. The method of claim 3 , wherein the first algorithm comprises multiple layers of processing, and the portion of the data stream to be stored is determined based on the output of an upper layer of the first algorithm.

5. 5. The method of claim 1, wherein the first algorithm and the second algorithm are trained using a common training data set and a common training objective.

6. 6. The method of claim 1, further comprising the step of pre-processing one or more portions of the data stream to be stored in the first buffer before storing the determined portions in the first buffer.

7. determining one or more portions of the data stream to store in a first buffer includes determining a substantially fixed percentage of the data stream to store in the first buffer based on an output of the upper layer of the first algorithm; Each activation in the upper layer is represented as a point in a multidimensional space; The method of claim 4 , wherein the portion determined to be removed from the first buffer is removed based on a distance metric between points in the multi-dimensional space.

8. 8. The method of claim 1, further comprising updating the first algorithm based on the identification of one or more second content characteristics by the second algorithm.

9. storing the one or more first content characteristics identified by the first algorithm in a second buffer; 9. The method of claim 1, wherein determining one or more portions of the data stream to store in the first buffer is based at least in part on characteristics of first content stored in the second buffer.

10. the data stream is a video stream including a plurality of image frames; The method of claim 1 , wherein the first content feature and the second content feature are image features of the image frame.

11. the image features correspond to people; The method of claim 10 , wherein the generated output is based on an estimated number of people in the video stream.

12. the image features correspond to one or more diagnostic indicators; The method of claim 10 , wherein the generated output includes at least one diagnostic indicator to aid in diagnosis.

13. 1. A data processing device, comprising: an input for receiving an input data stream; a processor; Output and A data processing apparatus comprising: a processor configured to perform a method according to any one of claims 1 to 12;

14. A storage medium configured to store instructions which, when executed by a processor, cause the processor to perform the method of any of claims 1 to 12.

Images