Video camera and method for analyzing video streams

Abstract

A video camera and method for analyzing a video stream are disclosed. The video camera includes: an image sensor that captures image data of image frames of the video stream; an image buffer that buffers the image data of the image frames; and a circuit. The circuit is configured to execute: a first evaluation function that evaluates a subset of image frames in the video stream by evaluating data related to the subset of image frames and outputs a corresponding first evaluation result for each evaluated image frame, wherein the evaluation of the image frame has an evaluation time greater than 1 / fps; a buffer function that marks the image frame as being evaluated for each image frame evaluated by the first evaluation function and instructs the image buffer to buffer the image data of the marked image frame; and a second evaluation function that accesses the image buffer to evaluate the image data of each image frame in the subset of image frames based on the corresponding first evaluation result when the corresponding first evaluation result is available, and outputs the corresponding second evaluation result.

Description

Technical Field

[0001] The present invention relates to a video camera and a method for analyzing a video stream using a series of evaluation functions, and in particular, to the use of image buffers during the running of a video analysis using such a series of evaluation functions. Background Art

[0002] Video analysis can be performed using a series of evaluation functions. For example, a first evaluation function Algo1 is dedicated to performing a first evaluation, such as video stream terminal face detection, and a second evaluation function Algo2 is dedicated to performing a second evaluation, such as facial recognition on a face detected by Algo1. Typically, the output from Algo1 consists of metadata (such as the timestamp of the video frame in the video stream where the face was detected and the coordinates of the face in the video frame). Algo2 is then able to identify the video frame in the video stream by using the timestamp and crop the facial region (multiple) according to the coordinates given in the metadata and perform further analysis, such as facial recognition. Therefore, Algo1 needs to evaluate many frames of the video stream but Algo2 is only interested in a subset of frames that Algo1 has marked as important, in this example, marked as containing detected faces.

[0003] In order to ensure that Algo2 can evaluate image frames marked as important by Algo1, image data related to image frames up to a certain period of time needs to be buffered. This ensures that Algo2 can access image data related to image frames marked as important by Algo1. Typically, for various performance reasons, the image data related to the image frames needs to be raw image data. The amount of image data that needs to be buffered can become quite large, especially if Algo1 has a certain latency between the capture of image data and the delivery of metadata. According to one example, the latency between the capture of image data and the delivery of metadata is 250ms, and the image data is captured at a rate of 60 frames per second (fps). According to this example, at least 15 (60fps*0.25s) full-resolution raw images need to be buffered. If the latency of Algo1 increases, even more image data needs to be buffered.

[0004] However, especially for video cameras, the size of the image buffer, in which the above mentioned video analysis is typically performed, is often limited.Therefore, there is a need to develop video analysis comprising a series of analysis functions. Summary of the invention

[0005] An object is to mitigate, alleviate or eliminate one or more of the above mentioned deficiencies in the prior art and the disadvantages singly or in any combination, and to at least partially solve the above mentioned problems.

[0006] According to a first aspect, a video camera is provided. The video camera is configured to analyze a video stream. In particular, the video camera is configured to analyze the video stream using a series of evaluation functions. The video camera includes an image sensor, an image buffer, and a circuit. The image sensor is configured to capture image data of an image frame of the video stream. The image buffer is configured to buffer the image data of the image frame. The circuit is configured to execute a first evaluation function, a buffer function, and a second evaluation function. The first evaluation function is configured to evaluate a subset of image frames in the video stream by evaluating data related to the subset of image frames. The first evaluation function is configured to output a corresponding first evaluation result for each evaluated image frame. The evaluation of the image frame has an evaluation time greater than 1 / fps, where fps is the fps of the video stream. The buffer function is configured to mark the image frame as being evaluated for each image frame evaluated by the first evaluation function and instruct the image buffer to buffer the image data of the marked image frame. The second evaluation function is configured to access the image buffer to evaluate the image data of each image frame in the subset of image frames based on the corresponding first evaluation result when the corresponding first evaluation result is available. The second evaluation function is configured to output the corresponding second evaluation result.

[0007] Note that in a series of evaluation functions, the first evaluation function has a latency greater than 1 / fps, which is the fps of the video stream, and the second evaluation function requires the result from the first evaluation function in order to perform its evaluation. Image buffer storage can be saved through the inventors' insight into how to buffer image data during the execution of video analysis. In particular, during the execution of video analysis, a series of evaluation functions are used. By indicating which image frames of the video stream to evaluate for the first evaluation function, it is allowed to buffer image data for only the image frames that are being evaluated. Therefore, memory usage of the image buffer can be saved.

[0008] The image buffer may be configured to initially buffer image data for all captured image frames. The buffering function may be configured to, when instructing the image buffer to buffer image data for a particular image frame of a subset of image frames, instruct the image buffer to delete image data for image frames that are not marked as being evaluated and that are temporally preceding the particular image frame.

[0009] The buffer function may be configured to instruct the image buffer to delete image data of an image frame corresponding to the output second evaluation result.

[0010] The data relating to the subset of image frames may include one or more of: image data for the subset of image frames, and other sensor data corresponding in time to the subset of image frames.

[0011] The video camera may include a transceiver configured to receive other sensor data.

[0012] The video camera may include one of a plurality of sensors configured to capture other sensor data.

[0013] The first evaluation function may include one or more of the following: an object detection function, a motion detection function, a change detection function, and an image segmentation function.

[0014] The second evaluation function may include one or more of the following: a target recognition function and an object classification function.

[0015] The first evaluation result may include an identification of a region of the corresponding image data to be evaluated by the second evaluation function.

[0016] The buffering function may be configured to mark image frames of the subset of image frames as being subject to evaluation by adding data to headers of the image frames.

[0017] The buffer function may be configured to mark image frames of the subset of image frames as being evaluated by updating a register on image frames that were evaluated by the first evaluation function.

[0018] According to a second aspect, a method for analyzing a video stream comprising a plurality of image frames is provided. The method is a video analysis method. A series of evaluation functions may be used to perform the video analysis. The method comprises:

[0019] The first evaluation function evaluates a subset of image frames in the video stream by evaluating data related to the subset of image frames, and outputs a corresponding first evaluation result for each evaluated image frame, wherein the evaluation of the image frame has an evaluation time greater than 1 / fps, where fps is the fps of the video stream;

[0020] for each image frame evaluated by the first evaluation function, marking the image frame as being evaluated and instructing the image buffer to buffer image data of the marked image frame;

[0021] evaluating, by a second evaluation function configured to access the image buffer, image data of each image frame in the subset of image frames based on the corresponding first evaluation result when the corresponding first evaluation result is available; and

[0022] A corresponding second evaluation result is output for each image frame in the subset of image frames.

[0023] The method may include: initially buffering, by the image buffer, image data of all image frames in the video stream when capturing image data thereof. The step / action of instructing the image buffer to buffer image data of particular image frames of the subset of image frames includes: instructing the image buffer to delete image data of image frames that are not marked as being evaluated and that are temporally preceding the particular image frame.

[0024] Where applicable, the above mentioned features of the video camera also apply to this second aspect.In order to avoid excessive repetition, reference is made to the above.

[0025] According to a third aspect, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium has instructions stored thereon, the instructions being used to implement the method according to the second aspect when the instructions are executed on a device having processing capabilities. The device may be a video camera, for example, the video camera of the first aspect. The device may be a video management server.

[0026] According to the specific embodiments given below, the further scope of application of the present invention will become apparent. However, it should be understood that the specific embodiments and specific examples are given as illustrations only when indicating the preferred embodiments of the present invention, because according to the specific embodiments, various changes and modifications within the scope of the present invention will become apparent to those skilled in the art.

[0027] Therefore, it should be understood that the present invention is not limited to the specific components of the described devices or the behavior of the described methods, because such devices and methods can be changed. It will also be understood that the terms used herein are only used for the purpose of describing specific embodiments and are not intended to be used for limitation. It must be pointed out that, as used in the specification and the appended claims, the articles "one", "the" and "said" are intended to mean that there are one or more elements, unless the context clearly indicates otherwise. Therefore, for example, a reference to a "unit" or "the unit" can include several devices, etc. In addition, the words "comprise", "comprises", "comprising", "including" and similar expressions do not exclude other elements or steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other aspects of the invention will now be described in more detail with reference to the attached drawings.The drawings should not be considered limiting; rather they are for explanation and understanding.

[0029] As illustrated in the figures, the sizes of layers and regions may be exaggerated for illustrative purposes and, therefore, are provided to illustrate the general structure. Like reference numerals refer to like elements throughout.

[0030] FIG. 1 schematically illustrates latency and image buffer usage for video analysis using a series of evaluation functions according to the prior art.

[0031] Figure 2 Schematically illustrating latency and image buffer usage for video analysis using a series of evaluation functions according to the present invention.

[0032] Figure 3is a block diagram of a method for analyzing a video stream including a plurality of image frames.

[0033] Figure 4 The schematic diagram illustrates a video camera configured to perform video analysis using a series of evaluation functions. DETAILED DESCRIPTION

[0034] The present invention will now be described more fully below with reference to the accompanying drawings in which currently preferred embodiments of the present invention are shown. However, the present invention may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided for the sake of thoroughness and completeness, and to fully convey the scope of the present invention to those skilled in the art.

[0035] With respect to FIG. 1 , the use of an image buffer for video analysis using a series of evaluation functions according to the prior art will be discussed. As mentioned in the background section, a series of evaluation functions can be used to perform video analysis. Let us assume that a series of evaluation functions includes a first evaluation function and a second evaluation function. The second evaluation function can only be executed when the first evaluation function is ready. Therefore, the second evaluation function depends on the result from the first evaluation function. In the example to be discussed with respect to FIG. 1 , the image frames and α, β, and of the video stream are evaluated by the first evaluation function. The evaluation of image frame α begins at time A and the result of the evaluation performed by the first evaluation function is ready at time A'. The evaluation of image frame β begins at time B and the result of the evaluation performed by the first evaluation function is ready at time A'. The evaluation of image frame β begins at time C and the result of the evaluation performed by the first evaluation function is ready at time C'. This is illustrated in the upper figure of FIG. 1 , in which a video stream is represented by a plurality of sequential image frames arranged along a timeline. Image frames α, β, and are marked in the video stream. In addition, different times A, A', B, B', C, and C' are marked on the timeline. In Figure 1, the waiting time of the first evaluation function is equally long for each evaluation performed. Therefore, the waiting times A-A', B-B' and C-C' are equally long. It should be noted that the evaluations performed by the first evaluation function have an evaluation time greater than 1 / fps, where fps is the fps of the video stream.

[0036] In order to be executed, the second evaluation function needs to have access to image data related to the image frames evaluated by the first evaluation function and the results of the evaluation performed by the first evaluation function. Typically, the second evaluation function needs to have access to the original image data, or at least uncompressed image data related to the image frames evaluated by the first evaluation function. Therefore, in order to ensure that the second evaluation function can access the image data related to the image frames evaluated by the first evaluation function, the image data related to the image frames until a certain period of time needs to be buffered in the image buffer. This will be explained in detail in the lower part of Figure 1. The image data related to the image frames are continuously buffered in the image buffer. In Figure 1, the use of the image buffer at times A', B' and C' will be discussed.

[0037] Until time A', image data relating to all image frames before A' have been continuously buffered in the image buffer. This is illustrated in the "A':" diagram in the lower portion of Figure 1. At time A', the evaluation of image frame α by the first evaluation function is completed. All image data relating to image frames before image frame α can then be discarded, i.e., image data relating to image frames before time A can be removed from the image buffer. However, all image data relating to image frames after time A must still be maintained in the image buffer. This is because it is still unknown what image data after time A the second evaluation function needs to have access to. Therefore, all image data relating to image frames after time A need to be buffered until new results from the first evaluation function are ready.

[0038] From time A to time B', image data relating to image frames intermediate between these two times needs to be buffered, which is illustrated in the lower portion of FIG. 1 at "B':". At B', the evaluation of image frame β by the first evaluation function is completed. All image data relating to image frames prior to image frame β can then be discarded, i.e., image data relating to image frames prior to time B can be removed from the image buffer. There is an exception to this, and image data relating to image frame α may still need to be in the image buffer. This is because the second evaluation function may not have yet completed evaluating α. However, once the second evaluation function completes its evaluation of α, image data relating to image frame α can be removed from the image buffer. In addition, all image data relating to image frames after time B must still be maintained in the image buffer. This is because it is still unknown what image data after time B the second evaluation function needs to have access to. Therefore, all image data relating to image frames after time B need to be buffered until new results from the first evaluation function are ready.

[0039] From time B to time C', image data related to image frames between these two times need to be buffered, which is illustrated in the lower part of Figure 1 at "C':". At time C', the evaluation of the image frame by the first evaluation function is completed. All image data related to the image frame before the image frame can then be discarded, that is, the image data related to the image frame before time B can be removed from the image buffer. There is an exception to this, and the image data related to image frames α and β may still need to be located in the image buffer. This is because the second evaluation function may not have completed the evaluation of α and / or β. However, once the second evaluation function completes its evaluation of α and / or β, the image data related to image frames α and / or β can be removed from the image buffer. In addition, all image data related to image frames after time C must still be kept in the image buffer. This is because it is still unknown what image data after time C the second evaluation function needs to have access to. Therefore, all image data related to image frames after time C need to be buffered until the new results from the first evaluation function are ready.

[0040] Now turn to Figure 2 .about Figure 2 , the use of image buffers for video analysis using a series of evaluation functions according to the present invention will be discussed. Again we assume that a series of evaluation functions include a first evaluation function and a second evaluation function. The second evaluation function can only be executed when the first evaluation function is ready. Therefore, the second evaluation function depends on the result from the first evaluation function, that is, the first evaluation result output from the first evaluation function. Data related to image frames α, β and of the video stream are evaluated by the first evaluation function. Data related to image frames α, β and of the video stream can be image data related to corresponding image frames α, β and. Such image data can be raw image data, or at least uncompressed image data. Alternatively, or in combination, data related to image frames α, β and can be data from another type of sensor. Such data will now be referred to as sensor data. Examples of sensors that can be used to generate such sensor data are RADAR sensors, LIDAR sensors, PIR sensors and / or microphones. Data related to image frames analyzed by the first evaluation function are captured at the same time as the image data of the corresponding image frame. Therefore, the data evaluated by the first evaluation function is temporally associated with the image data of the corresponding image frame. Thus, the data evaluated by the first evaluation function may be the same image data evaluated by the second evaluation function. Alternatively, or in combination, the data evaluated by the first evaluation function may be other sensor data. The other sensor data is typically temporally associated with the image data to be evaluated by the second evaluation function.

[0041] The evaluation of the data related to the image frame α by the first evaluation function begins at time A and the results from the evaluation are ready at time A'. The evaluation of the data related to the image frame β by the first evaluation function begins at time B and the results from the evaluation are ready at time B'. The evaluation of the data related to the image frame by the first evaluation function begins at time C and the results from the evaluation are ready at time C'. Figure 2 The upper figure of FIG. 1 shows a video stream represented by a plurality of sequential image frames arranged along a timeline. Image frames α, β and α are marked in the video stream. In addition, different times A, A', B, B', C and C' are marked on the timeline.

[0042] exist Figure 2 , the waiting time of the first evaluation function is equally long for each evaluation performed. Therefore, the waiting times A-A', B-B' and C-C' are equally long. However, please note that the waiting time of the first evaluation function can vary. The variation of the waiting time of the first evaluation function can vary, for example, due to the composition of the input data to the first evaluation function. It should be noted that the evaluation performed by the first evaluation function has an evaluation time greater than 1 / fps, where fps is the fps of the video stream.

[0043] refer to Figure 3 , a block diagram of a method 300 for analyzing a video stream including a plurality of image frames will be initially discussed. The method comprises: evaluating a subset of image frames in a video stream S302 by a first evaluation function. Wherein, the first evaluation function is configured to evaluate a subset of image frames in a video stream. The evaluation made by the first evaluation function is performed by evaluating data related to a subset of image frames. The data related to the subset of image frames may be image data of the corresponding image frames. The data related to the subset of image frames may be sensor data temporally associated with the corresponding image frames. The sensor data may be generated, for example, by a RADAR sensor, a LIDAR sensor, a PIR sensor and / or a microphone. Upon completion of the evaluation S302 of the data related to the image frames in the video stream, a first evaluation result corresponding to S306 is output. Therefore, the first evaluation function is configured to output a corresponding first evaluation result for each image frame being evaluated. The evaluation of the data related to the image frames performed by the first evaluation function has an evaluation time greater than 1 / fps, where fps is the fps of the video stream.

[0044] With respect only to the prior art solutions for use with the image buffer, the second evaluation function according to the solution of the present invention needs to have access to image data relating to the image frame evaluated by the first evaluation function and to the first evaluation result from the evaluation performed by the first evaluation function in order to be executed. Typically, the second evaluation function needs to have access to the original image data, or at least the uncompressed image data relating to the image frame evaluated by the first evaluation function. Therefore, the image data relating to the image frame evaluated by the first evaluation function needs to be buffered in the image buffer. Return to Figure 3 This is done by, for each image frame evaluated by the first evaluation function, marking the image frame as being evaluated and instructing S304 the image buffer to buffer image data of the marked image frame.

[0045] Additionally, the image buffer may be configured to initially buffer image data of all image frames in the video stream when all image frames in the video stream are captured. In doing so, the step / action of instructing S304 the image buffer to buffer image data of a particular image frame of the subset of image frames comprises instructing the image buffer to delete image data of image frames that are not marked as being evaluated and that are temporally preceding the particular image frame.

[0046] Thus, the image buffer is instructed to "only" buffer image data corresponding to the image frame evaluated by the first evaluation function. Figure 2 The lower figure of is shown. At the beginning of the evaluation by the first evaluation function, the image buffer is instructed to buffer image data belonging to the image frame being evaluated. Therefore, at time A, the image buffer is instructed to buffer image data belonging to image frame α. At time B, the image buffer is instructed to buffer image data belonging to image frame β. At time C, the image buffer is instructed to buffer image data belonging to image frame. This Figure 2 . It should be noted that at time A, image data for image frame α is buffered. However, at time B, image data for both image frames α and β are buffered. This is because the second evaluation function may not have yet completed evaluating α. In addition, at time C, image data for image frames α, β, and are buffered. This is because the second evaluation function may not yet have completed evaluating α and / or β. However, once the second evaluation function completes its evaluation of image data associated with the image frame, such image data may be removed from the image buffer.

[0047] As in Figure 2As shown in the lower right portion of the diagram, at time A', the image buffer can hold image data for one additional image frame (i.e., the image frame captured at time A'). This is because the image buffer can be configured to initially buffer its image data when all image frames in the video stream are captured. However, at this point, the step / action of instructing S304 the image buffer to buffer image data for a particular image frame of a subset of image frames includes: instructing the image buffer to delete image data for image frames that are not marked as being evaluated and that are temporally preceding the particular image frame. Therefore, at time B', the image buffer will hold image data for one additional image frame (i.e., the image frame captured at time B'), and at time C', the image buffer will hold image data for one additional image frame (i.e., the image frame captured at time C'). Please note that the amount of image data that needs to be buffered is substantially reduced compared to the prior art solution illustrated with respect to FIG. 1 .

[0048] Again, return to Figure 3 , the image data of each image frame in the subset of image frames evaluated by the first evaluation function and buffered in the image buffer is evaluated S308 by the second evaluation function. Therefore, the second evaluation function is configured to access the image buffer to evaluate the image data of each image frame in the subset of image frames based on the corresponding first evaluation result when the corresponding first evaluation result is available. When the second evaluation function prepares the evaluated image data of the image frame, it can be further configured to output S310 the result as the corresponding second evaluation result.

[0049] The method 300 may further include: when outputting S310 the second evaluation result, removing S312 image data of the image frame evaluated by the second evaluation function from the image buffer.

[0050] about Figure 4 , will discuss the above Figure 2 and 3 The video camera 400 discussed is configured to perform a series of evaluation function video analysis. Therefore, the video camera 400 is configured to analyze the video stream captured by it. The video camera 400 can be a surveillance camera. The video camera 400 can be fixed. The video camera 400 can be a video camera installed on a moving object such as a vehicle. The video camera 400 may be damaged by a person. The video camera 400 includes an image sensor 402, an image buffer 410 and a circuit 420.

[0051] The image sensor 402 is configured to capture image data of image frames of a video stream. Image sensors and the capture of image data are well known to those skilled in the art and will not be discussed in any further detail in the present disclosure.

[0052] Image buffer 410 is configured to buffer image data of image frames. Image buffer 410 is typically an area or part of a physical storage device for temporarily storing image data when the image data is processed. Typically, the image data is stored in image buffer 410 as captured by image sensor 402. However, image buffer 410 can be used when moving image data between functions. Image buffer 410 can be implemented in a fixed memory location in hardware or by using a virtual data buffer in software pointing to a location in a physical storage device. In any case, the image data stored in image buffer 410 is stored on a physical storage medium. Image buffer 410 can be implemented in software, which typically uses random access memory RAM to temporarily store image data. Due to the much faster access time compared to a hard drive, using RAM can be beneficial. Image buffer is typically used when there is a difference between the rate at which image data is received and the rate at which it can be processed.

[0053] The video camera 400 may further include one or more of a video encoder 430 , a transceiver 440 , and a memory 450 .

[0054] The video encoder 430 is configured to encode the image data of the image frames captured by the image sensor 402 into an encoded video stream. The video encoder 430 is configured to encode some of the images of the video stream as key frames. The key frame is an encoded video frame that does not require information from other encoded video frames in order to be decoded. Therefore, the key frame is encoded based on the information of the image from the video data to which the key frame is set. Typically, the image is encoded as a key frame using similarities within the image. In video encoding, a key frame is also called an intraframe, often referred to as an I frame. The video encoder 430 is further configured to encode the image of the video stream between two key frames as an incremental frame. Typically, an incremental frame only includes changes from one frame to the next frame. Therefore, compared with a key frame, an incremental frame typically includes less data. In video encoding, an incremental frame is also called an interframe, often referred to as a P frame or a B frame. A P frame refers to a previous frame used for data reference. Therefore, the content of the previous frame must be known in order to decode the P frame. A B frame can refer to both the previous frame and the previous frame used for data reference. Therefore, the contents of both the previous frame and the previous frame must be known in order to decode a B frame.

[0055] The transceiver 440 is configured to receive information from the video camera 400 and / or send information to the video camera 400. The transceiver 440 can be configured to send out an encoded version of the video stream. However, the transceiver 440 can be configured to also receive and / or transmit other information / data. For example, the transceiver 440 can be configured to receive sensor data from a sensor. Examples of sensors are laser sensors, LIDAR, sensors, PIR sensors, and microphones.

[0056] The memory 450 may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, a random access memory (RAM), or another suitable device. For example, the memory 450 may include the image buffer 410. In a typical arrangement, the memory 450 may include a non-volatile memory for long-term data storage and a volatile memory used as a system memory for the circuit 420. The memory 450 may exchange data with the circuit 420 via a data bus. There may also be associated control lines and an address bus between the memory 450 and the circuit 420. The memory 450 may further be configured to store an encoded version of the video stream and / or metadata generated by the video camera 400.

[0057] The circuit 420 is configured to perform the overall functions of the video camera 400. The circuit 420 may include a processor 422, such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 422 is configured to execute program codes stored in the memory 450 to perform the functions of the video camera 400.

[0058] The functionality of the video camera 400 may be embodied in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored on a non-transitory computer-readable medium (e.g., memory 450) of the video camera 400 and executed by the circuit 420 (e.g., using the processor 422). In addition, the functionality of the video server 400 may be a stand-alone software application or form part of a software application that performs additional tasks related to the video camera 400. The described functionality may be considered to be a method that the corresponding device is configured to perform. Moreover, when describing functionality that may be implemented in software, such functionality may also be performed via dedicated hardware or firmware, or some combination of hardware, firmware and / or software.

[0059] The circuit 420 is configured to execute a first evaluation function 452 , a buffer function 454 , and a second evaluation function 456 .

[0060] The first evaluation function 452 is configured to evaluate a subset of image frames in a video stream. This is done by evaluating data related to a subset of image frames. The data related to an image frame in a subset of image frames may be image data of a particular image frame. However, alternatively, or in combination, the data related to an image frame in a subset of image frames may be sensor data. Sensor data may be generated by any type of sensor. Examples of sensors that may be used to generate such sensor data are RADAR sensors, LIDAR sensors, PIR sensors, and / or microphones. The sensor may be outside the video camera 400. In this case, the transceiver 440 may be configured to receive other sensor data. Alternatively, or in combination, a sensor may be included in the video camera 400, which may include one or more sensors 460. In this case, the sensor 460 of the video camera 400 may be configured to generate sensor data.

[0061] Data related to the image frame analyzed by the first evaluation function 452 is captured at the same time as the image data of the corresponding image frame. Therefore, the data evaluated by the first evaluation function 452 is temporally associated with the image data of the corresponding image frame. Therefore, the data evaluated by the first evaluation function 452 can be the same image data that will be evaluated by the second evaluation function 456 later. Alternatively, or in combination, the data evaluated by the first evaluation function 452 can be other sensor data. Other sensor data is typically temporally associated with the image data to be evaluated by the second evaluation function 456. The evaluation of the image frame has an evaluation time greater than 1 / fps, which is the fps of the video stream. According to a non-limiting example, the evaluation time is 250ms and the video stream is captured at 60fps. According to this example, at least 15 (60fps*0.25s) image frames will be generated once the evaluation of the first evaluation function is performed.

[0062] The first evaluation function 452 is further configured to output a corresponding first evaluation result for each image frame being evaluated. The first evaluation result may include an identification of a region of the corresponding image data to be evaluated by the second evaluation function 456 .

[0063] The first evaluation function 452 may include one or more of the following: an object detection function, a motion detection function, a change detection function, and an image segmentation function.

[0064] The buffer function 454 is configured to mark the image frame as being evaluated for each image frame evaluated by the first evaluation function 452 and instruct the image buffer 410 to buffer the image data of the marked image frame. The image buffer 410 may be configured to initially buffer the image data of all captured image frames. The buffer function 454 may be configured to instruct the image buffer 410 to delete the image data of the image frame that is not marked as being evaluated in time before the specific image frame when instructing the image buffer 410 to buffer the image data of the specific image frame of the subset of image frames. The buffer function 454 may be configured to mark the image frame of the subset of image frames as being evaluated by adding data to the header of the image frame. Alternatively, or in combination, the buffer function 454 may be configured to mark the image frame of the subset of image frames as being evaluated by updating a register on the image frame evaluated by the first evaluation function 452.

[0065] The second evaluation function 456 is configured to access the image buffer 410 to evaluate the image data of the image frame evaluated by the first evaluation function 452 based on the corresponding first evaluation result when the corresponding first evaluation result is available. The second evaluation function 456 is further configured to output the corresponding second evaluation result. The second evaluation function 456 may include one or more of the following items: an object recognition function and an object classification function.

[0066] The buffer function 454 may be configured to instruct the image buffer 410 to delete image data of an image frame corresponding to the output second evaluation result.

[0067] Those skilled in the art realize that the present invention is not limited in any way to the preferred embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims.

[0068] For example, the video management server may execute Figure 2 and 3 The video management server may be implemented as a single server device or it may be implemented as a distributed server distributed over many different devices. The video management server may include an image buffer and circuits similar to the image buffer and circuits discussed with respect to video camera 400.

[0069] Please also note that in Figure 2 In the example embodiment, the evaluations made by the first evaluation function are non-periodic. However, for some applications, the evaluations made by the first evaluation function may be periodic.

[0070] Additionally, variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

Claims

1. A video camera for analyzing a video stream, the video camera comprising: an image sensor configured to capture image data of image frames of the video stream; an image buffer configured to buffer image data of an image frame; as well as The circuit is configured to perform: a first evaluation function configured to evaluate a subset of image frames in the video stream by evaluating data related to the subset of image frames, and configured to output a corresponding first evaluation result for each evaluated image frame, wherein the evaluation of the image frames has an evaluation time greater than 1 / fps, the fps being the fps of the video stream; a buffering function configured to, for each image frame evaluated by the first evaluation function, mark the image frame as being evaluated and instruct the image buffer to buffer image data of the marked image frame; and A second evaluation function is configured to access the image buffer to evaluate the image data of each image frame in the subset of image frames based on the corresponding first evaluation result when the corresponding first evaluation result is available, and output a corresponding second evaluation result.

2. The video camera according to claim 1, wherein: The buffer function is configured to instruct the image buffer to delete image data of an image frame corresponding to the output second evaluation result.

3. The video camera according to claim 1, wherein: The data relating to the subset of image frames comprises one or more of: image data for the subset of image frames and other sensor data corresponding in time to the subset of image frames. The video camera of claim 1 , further comprising a transceiver configured to receive other sensor data. 5 . The video camera of claim 1 , further comprising a sensor of the plurality of sensors configured to capture additional sensor data.

6. The video camera of claim 1, wherein: The first evaluation function includes one or more of the following: an object detection function, a motion detection function, a change detection function, and an image segmentation function.

7. The video camera of claim 1, wherein: The second evaluation function includes one or more of the following: an object recognition function and an object classification function.

8. The video camera of claim 1, wherein: The first evaluation result includes an identification of a region of corresponding image data that is to be evaluated by the second evaluation function.

9. The video camera of claim 1, wherein: The buffering function is configured to mark image frames of the subset of image frames as being subject to evaluation by adding data to headers of the image frames.

10. The video camera of claim 1, wherein: The buffer function is configured to mark image frames of the subset of the image frames as being evaluated by updating a register across image frames that are evaluated by the first evaluation function.

11. A method for analyzing a video stream comprising a plurality of image frames, the method comprising: evaluating the subset of image frames in the video stream by evaluating data related to the subset of image frames in the video stream by a first evaluation function, and outputting a corresponding first evaluation result for each evaluated image frame, wherein the evaluation of the image frames has an evaluation time greater than 1 / fps, where fps is the fps of the video stream; For each image frame evaluated by the first evaluation function, marking the image frame as being evaluated and instructing an image buffer to buffer image data of the marked image frame; evaluating, by a second evaluation function configured to access the image buffer, image data of each image frame in the subset of image frames based on the corresponding first evaluation result when the corresponding first evaluation result is available; and A corresponding second evaluation result is output for each image frame in the subset of image frames.

12. The method according to claim 11, wherein: The data relating to the subset of image frames comprises one or more of: image data for the subset of image frames and other sensor data corresponding in time to the subset of image frames.

13. A non-transitory computer-readable storage medium having stored thereon instructions for implementing the method of claim 11 when executed on a device having processing capabilities.

Images