Verification of updated analytical procedures in monitoring systems

Abstract

The present application discloses verification of an updated analysis program in a monitoring system. Specifically, a method (100) for evaluating an updated analysis program in a monitoring system is provided, the monitoring system including a plurality of monitoring devices arranged to monitor similar environments. The method comprises: identifying (S110) available processing resources, selecting (S120) a first monitoring device for which available processing resources have been identified, selecting (S130) a second monitoring device, obtaining (S140) monitoring data by the second monitoring device, and executing (S150) a current analysis program on the monitoring data. The method also comprises sending (S155) the monitoring data to the first monitoring device, executing (S160) an updated analysis program on the monitoring data in the first monitoring device, and evaluating (S190) the updated analysis program based on the results of the current analysis program and the updated analysis program.

Description

Technical Field

[0001] The present invention relates to the evaluation of updated analysis programs in a monitoring system comprising a plurality of monitoring devices. Background Art

[0002] Surveillance systems, including multiple monitoring / surveillance devices (such as cameras, radar devices, and audio devices), are widely used to gain awareness of objects and events within a particular environment or scene. The surveillance devices in such multi-device systems can be geographically distributed to allow, for example, detection of unauthorized access and criminal activity, as well as tracking of movement within the monitored environment or scene.

[0003] Surveillance data captured by surveillance equipment typically undergoes computer-assisted analysis programs designed to detect and identify objects, sounds, motion, and changes in the monitored environment or scene. Preferably, the analysis program is defined by software developed specifically for this purpose. However, developers of such software are known to face the challenge of testing and validating new versions to identify and correct software glitches. Software glitches can occur due to programmer errors or mistakes, resulting in bugs in the software code. If these glitches are implemented, the system may produce erroneous results or even cease to function, leading to system failure or downtime.

[0004] Therefore, testing newly developed things is common practice. Ideally, new developments are tested under as many combinations of inputs and prerequisites as possible. However, a fundamental challenge of software testing is that it is often not feasible to test under all combinations of inputs, states, and hardware types. Therefore, developers use strategies to select tests that are feasible within the available time and resources.

[0005] One strategy is to have new versions of the software tested by a selected subset of users, such as a small set of installed monitoring systems. If the performance is satisfactory, the new version can be rolled out to the remaining users or systems. While this strategy can provide some information about the quality of the software being tested, it is still difficult to ensure that the testing is sufficient to avoid failures and system downtime, and it is difficult to ensure that the selected set of monitoring / surveillance systems is representative of all possible environments and hardware types. Examples of the prior art are disclosed in US 2020 / 0159685 A1, which relates to a coordinated component interface control framework for deterministically reproducing the behavior of a data processing pipeline, and in US 10,896,116 B1, which relates to the detection of performance regressions in software for controlling autonomous vehicles. Summary of the Invention

[0006] It is an object of the present invention to provide a method for evaluating an updated analysis program in a monitoring system comprising a plurality of monitoring devices, which overcomes or alleviates at least some of the problems of known methods.

[0007] The invention is defined in independent claims 1, 12 and 15. Thus, according to a first aspect, a method for evaluating an updated analysis program in a monitoring system is provided, wherein the system comprises a plurality of monitoring devices, at least some of which are arranged to monitor similar environments. The method comprises the following:

[0008] Identify available processing resources in the monitoring system;

[0009] selecting a first monitoring device from a plurality of monitoring devices, available processing resources for which the first monitoring device has been identified;

[0010] selecting a second monitoring device from a plurality of monitoring devices;

[0011] Acquiring monitoring data by a second monitoring device;

[0012] executing the current analysis procedure on the monitoring data to produce a first result;

[0013] sending the monitoring data to a first monitoring device;

[0014] executing the updated analysis program on the monitoring data in the first monitoring device to produce a second result;

[0015] calculating a first performance value based on the first result, the first performance value indicating the performance of the second monitoring device when executing the current analysis program;

[0016] calculating a second performance value based on the second result, the second performance value indicating the performance of the first monitoring device when executing the updated analysis program; and

[0017] The updated analysis procedure is evaluated based on a comparison between the first performance value and the second performance value.

[0018] Several advantages are associated with this aspect. First, by selecting a monitoring device for executing an updated analysis program based on available processing resources, excess resources can be used efficiently for evaluation. This allows the evaluation to be performed with a reduced impact on the overall performance of the system. Second, using the first monitoring device to execute the updated analysis program on monitoring data acquired by the second monitoring device allows the updated analysis program to be evaluated without interfering with or affecting the operation of the second monitoring device. Therefore, a first monitoring device, which may have more available resources than the second monitoring device, can be used to evaluate the updated analysis program without adding a burden to the second monitoring device. Third, the evaluation can be considered valid for some or all monitoring devices that are arranged to monitor an environment similar to that of the second monitoring device. In other words, this aspect allows an update to be tested in the first monitoring device using monitoring data acquired by the second monitoring device before the update is rolled out to the second monitoring device and / or other monitoring devices that monitor the same or similar environment as the second monitoring device.

[0019] Evaluating an updated analysis program generally means executing the analysis program with the goal of discovering faults and verifying its suitability for use. More specifically, this may involve verifying that no faults have been introduced into the updated version, and further verifying that the performance of the analysis program has not been compromised. As discussed in more detail below, in some examples, the performance of the analysis program may refer to the performance of the program itself, i.e., the quality of the program's results in terms of accuracy and reliability, and in some examples, the processing resources or time utilized.

[0020] A monitoring system may be used to keep an environment, such as a geographical location or area, under surveillance or observation.In this disclosure, the terms monitoring and surveillance may be used interchangeably.

[0021] An environment (or scene) can be understood as any physical location, area, or space monitored or "seen" by a monitoring device. Two monitoring devices monitoring the same (or at least partially the same) physical location can therefore acquire monitoring data representing the environment that may differ slightly (e.g., due to different perspectives and viewing distances) but is still considered similar because they originate (at least partially) from the same physical location, i.e., the same geographic area or space. In particular, an environment can be a scene imaged by a monitoring camera of a monitoring system, or an area monitored by a radar or an audio device of a radar or an audio system, respectively. In some embodiments, multiple monitoring devices can be geographically distributed and arranged to monitor the same physical location or object from different positions or angles, or at least partially overlap the same portion. Thus, these environments can be considered similar, as observed by each monitoring device. Alternatively, multiple monitoring devices can monitor different physical locations, objects, or audio environments. In this case, "similar environments" can be understood as locations or objects monitored by the monitoring devices that meet similarity requirements. For example, certain types of monitored environments can be considered similar, such as indoor environments or outdoor environments. Specific types of monitored indoor or outdoor environments, such as office environments, elevators, stairs, parking lots, driveways, etc. can also be considered similar. For example, the similarity can be determined using scene classification methods, which aim to classify a monitored area or images of such an area into one of a number of predefined scene categories. For example, D. Zheng et al. describe examples of such methods in "Deep Learning for Scene Classification: A Survey" in Computer Vision and Pattern Recognition, 2021. Thus, in one example, an environment classified as "elevator" might be considered similar to environments classified as "emergency exit" and "library", respectively. Scene classification can also be associated with optical properties, such as reference lighting, as well as the viewing angle and altitude at which the environment is monitored. Thus, the classification "elevator" with the additional feature "backlight" can define environments including backlit elevators as similar.

[0022] Because similar reasoning can be applied to monitored objects, monitored environments that include objects of a particular class or type (such as, for example, doors, cars, and display cases) can be considered to satisfy the similarity requirement. In the case where the monitoring system is an audio system, in some examples, the term "similar environment" can be understood as a monitored audio environment or acoustic environment (sometimes called a soundscape), or a set of acoustic resources that share certain common characteristics. For example, two audio environments can be considered similar if they have similar acoustic effects, or if the sounds originate from similar sources.

[0023] As an alternative or in addition to the above, it will be appreciated that environments or scenarios may be defined as similar by an operator or installer.

[0024] In the case where the surveillance system includes a camera system or a radar system, in some examples, the monitored environment can be referred to as a scene. A scene can be understood as any physical area or space whose size and shape is defined by the field of view (FOV) of the camera or radar device.

[0025] Surveillance data can be understood as a data set that describes specific characteristics or attributes associated with the monitored environment. Surveillance data can be acquired, for example, by a camera (e.g., where the surveillance system includes a camera system), a radar device (e.g., where the surveillance system includes a radar system), or an audio device (e.g., where the surveillance system includes an audio system), and thus can include optical information, radio wave information, or acoustic information, respectively. In some examples, surveillance data can be acquired at a specific point in time (e.g., a single image in the case of a camera system), or continuously over a specific period of time (e.g., a video sequence or an audio stream).

[0026] Available processing resources can be identified by monitoring actual processor utilization or processor load. Alternatively, or in addition, available processing resources can be estimated, for example, based on historical data or known utilization patterns. Furthermore, available processing resources can refer to current or future processor utilization. In the latter case, the updated analysis program can be scheduled for execution at a later point in time when it is determined or estimated that processing resources are available. In some examples, available processing resources can be defined as time periods when there is little or no interest in the monitored environment, or time periods when the physical location is less important for monitoring. For example, this can be understood as time periods with fewer events.

[0027] An analysis program can be understood as an executable logic routine (e.g., a line of code, a software program, etc.) that defines a method that a processing unit (e.g., a processor) can perform to determine information about surveillance data. The information may, for example, refer to objects or sounds identified in the monitored environment, or motion, events, or actions occurring in the monitored environment. Thus, the analysis program may involve, for example, object detection, object tracking, object recognition, object re-identification, object masking, motion recognition, and motion detection based on surveillance data acquired by, for example, a camera system or a radar system. Based on surveillance data acquired by, for example, an audio system, the analysis program may further involve voice detection, gunshot detection, sound triangulation, attack detection, and voice detection. The output of the analysis program may be referred to as a result. The results of the current and updated analysis programs can be analyzed to evaluate and verify the updated analysis programs on the surveillance data.

[0028] As already mentioned, the first performance value and the second performance value may indicate the time consumed to execute the current and updated analysis programs, respectively. In a further example, the performance value may indicate power consumption, i.e., the power (e.g., in watts) used by each monitoring device, processing device, or the entire system when executing the current and updated analysis programs, respectively. The performance value may further relate to the processing resources used to execute the analysis programs.

[0029] These data—time consumed, power resources used, or processing resources used—can be used as input when evaluating the updated analysis program. An increase in any of these measures may indicate that the updated analysis program is compromised compared to the current analysis program.

[0030] Alternatively or additionally, the first performance value and the second performance value may indicate the accuracy or reliability of the current analysis program and the updated analysis program, respectively. For example, the performance value may relate to the ability to correctly identify objects, actions, or sounds in the monitored environment, and an evaluation of the updated analysis program may help ensure that performance is not compromised by replacing the current analysis program with the updated analysis program.

[0031] Typically, the idea is to use a first monitoring device (for which available processing resources have been identified) to perform an updated analysis procedure on the monitoring data (possibly consuming resources). However, it will be understood that the remaining calculations and operations, such as the current analysis procedure, the calculation of the first performance value and the second performance value and the comparison of the two, can be performed elsewhere, for example by a second monitoring device or by a remotely located processing unit (also known as a central processing unit, central processor or central circuit), depending on the architecture of the monitoring system and the typical available processing resources. Therefore, the current analysis procedure can be performed by the second monitoring device or by the central processing unit on which the current analysis procedure has been installed. In addition, the calculation of the first performance value and the second performance value based on the results from the current and updated analysis procedures can be performed by the first monitoring device, the second monitoring device or the central processing unit, respectively.

[0032] Therefore, in one embodiment, the second monitoring device may execute the current analysis procedure on the monitoring data and send the first result to the first monitoring device for calculating the first performance value. The first monitoring device may further calculate the second performance value based on the second result.

[0033] In another embodiment, the first monitoring device may send the second result to the second monitoring device, and the second monitoring device may calculate the first performance value and the second performance value.

[0034] As indicated above, the evaluation of the updated analysis program includes a comparison between a first performance value and a second performance value, which can be performed by one of the first monitoring device and the second monitoring device or by a central processing unit, and the first performance value and the second performance value can be sent from the monitoring device that performs the calculation of the corresponding performance values ​​to the central processing unit.

[0035] If the evaluation indicates that there is an improvement, or at least that the updated analysis program meets the specifications and requirements and thus achieves its intended purpose, the updated analysis program can be sent to the second monitoring device and / or other monitoring devices used to monitor the same or similar environment as the second monitoring device, such as a third monitoring device. In other words, the updated analysis program can be extended to one or more of the remaining monitoring devices in the monitoring system after being verified and validated in the first monitoring device. The updated analysis program can be considered to have been verified and validated for all monitoring devices in the system that monitor the same or similar environment as the second monitoring device. As will be discussed further below, monitoring devices monitoring the same or similar environment can preferably have similar hardware types to further increase the reliability of the verification.

[0036] The transfer or upgrade of updated analytical programs can be performed automatically at a predetermined or scheduled point in time immediately after the evaluation is completed, or manually upon the user's request.

[0037] If the evaluation indicates that the updated analysis program does not meet specifications and requirements, the updated analysis program may be rolled back from the first monitoring device and replaced (eg, with the current analysis program) so as not to interfere with subsequent operations of the first monitoring device.

[0038] A monitoring device among a plurality of monitoring devices in the system that is determined to have sufficient processing resources to execute (and possibly evaluate) the updated analysis program can be selected as a first monitoring device. The required resources can be included as a specification or requirement submitted with the updated analysis program, such as by a developer or entity providing the update, or estimated based on the resources required to execute the current analysis program. In another example, the processing resources used can be monitored during execution of the updated analysis program and used as input for future updates.

[0039] Preferably, the first monitoring device and the second monitoring device may be of a similar type to ensure that the evaluation of the updated analysis program in the first monitoring device is also valid for the second monitoring device. This also applies to other monitoring devices on which the updated analysis program is to be evaluated. Therefore, the monitoring devices may be similar, of a similar type or include similar hardware. More specifically, similar monitoring devices may include similar mechanisms or types of mechanisms for executing the analysis program. Similar monitoring devices, such as for example the first monitoring device and the second monitoring device, may, for example, include similar hardware, i.e. a hardware accelerator, a central processing unit (CPU) or a graphics processing unit (GPU) for executing the analysis program. Examples of hardware accelerators include neural network (NN) accelerators, neural processors, deep learning processors (DLPs) and artificial intelligence (AI) accelerators.

[0040] The analysis procedure may be performed using a deep learning approach, which may be understood as a type of machine learning that may involve training a model, typically referred to as a deep learning model. Thus, the analysis procedure may be performed using a deep learning model that is configured and trained to perform the analysis procedure at hand. Viewed at a general level, the input to such a deep learning model may be monitoring or surveillance data to be analyzed (such as image data, radar data, or audio data), and the output may be a tensor representing confidence scores for one or more analysis results of the monitoring data at a data level. In the case where the monitoring data is image data and the analysis procedure is an object detection procedure, the output tensor may represent confidence scores for one or more object classes of the image data at a pixel level. In other words, the deep learning model may determine, for each pixel, the probability that the pixel depicts each object in one or more object classes.

[0041] After training the deep learning model, the analysis program can be considered to have been updated. For example, the training can be stimulated by access to new input data. The analysis program can be further considered to have been updated if the deep learning framework that can be used to design, train and interact with the deep learning model and the hardware has been modified or updated. This may occur, for example, if the framework provider releases a new version of the deep learning framework. In some examples, changes in the deep learning framework can stimulate new training of the deep learning model. The updated analysis program can therefore be understood as at least one of the deep learning model and the deep learning framework being updated, or at least different from the deep learning model and deep learning framework of the current analysis program. Therefore, according to some examples, the updated analysis program can be executed using a deep learning model operating in a deep learning framework, where at least one of the deep learning model and the deep learning framework has been updated.

[0042] It will be appreciated that in some embodiments, a monitoring system or surveillance system may be or include camera systems arranged to view similar scenes. Thus, a first monitoring device may include a first camera and a second monitoring device may include a second camera. Furthermore, in some examples, the monitored environment may be a scene defined by at least partially overlapping fields of view of the cameras. The surveillance data may therefore include images or image streams captured by the second camera.

[0043] Surveillance systems may also include light detection and ranging (LIDAR) systems that use lasers to determine distances and image the environment.

[0044] In some embodiments, the monitoring system may be or include a radar system including a first radar device as the first monitoring device and a second radar device as the second monitoring device. In this case, the monitoring data may include radio wave information.

[0045] Furthermore, in some embodiments, the monitoring system may be or include an audio system for monitoring sounds in an environment. The first monitoring device may therefore include a first audio device, the second monitoring device includes a second audio device, and the monitoring data includes sound information captured by the second audio device.

[0046] According to a second aspect, an evaluation system is provided that is configured to evaluate an updated analysis program in a surveillance system, such as a camera system, a radar system, or an audio system, the surveillance system comprising a plurality of surveillance devices, at least some of which are arranged to monitor similar environments. The evaluation system comprises circuitry configured to perform:

[0047] an identification function configured to identify available processing resources in the monitoring system,

[0048] a selection function configured to select a first monitoring device from a plurality of monitoring devices for which available processing resources have been identified, and further configured to select a second monitoring device from the plurality of monitoring devices, and

[0049] The monitoring data acquisition function is configured to acquire monitoring data using the second monitoring device. The circuit is further configured to perform:

[0050] a monitoring data sending function configured to send the monitoring data to the first monitoring device,

[0051] an analysis program execution function configured to execute a current analysis program on the monitoring data, thereby producing a first result, and further execute an updated analysis program on the monitoring data using the first monitoring device, thereby producing a second result,

[0052] a calculation function configured to calculate a first performance value indicating the performance of the second monitoring device when executing the current analysis program based on the first result, and further calculate a second performance value indicating the performance of the first monitoring device when executing the updated analysis program based on the second result, and

[0053] An evaluation function is configured to evaluate the updated analysis program based on a comparison between the first performance value and the second performance value.

[0054] According to a third aspect, there is provided a non-transitory computer-readable medium having computer code instructions stored thereon, the computer code instructions being adapted to perform the method of the first aspect when executed by a device having processing capabilities (such as the circuit discussed in conjunction with the second aspect).

[0055] The above optional additional features of the method according to the first aspect also apply to the second and third aspects, where applicable. To avoid unnecessary repetitions, reference is made to the above.

[0056] Further scope of applicability of the present invention will become apparent from the detailed description given below. However, it should be understood that the detailed description and specific examples showing preferred embodiments of the present invention are given by way of illustration only, as various changes and modifications within the scope of the present invention will become apparent to those skilled in the art from this detailed description.

[0057] Therefore, it should be understood that the present invention is not limited to the specific acts of the methods described, as these methods may vary. It should also be understood that the terminology used herein is for the purpose of describing specific embodiments only and is not restrictive. It must be noted that in the specification and the appended claims, the articles "a", "an", "the" and "said" are intended to indicate the presence of one or more elements, unless the context clearly dictates otherwise. Thus, for example, reference to "a unit" or "the unit" may include several devices, etc. In addition, "comprises", "comprising", "containing" and similar expressions do not exclude other elements or steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The above and other aspects of the present invention will now be described in more detail with reference to the accompanying drawings. The accompanying drawings should not be considered limiting but are used to explain and understand the present invention as defined in the appended claims. The same reference numerals refer to the same elements throughout.

[0059] Figure 1 An exemplary evaluation system for evaluating updated analytical procedures in a monitoring system is shown in accordance with some embodiments.

[0060] Figure 2a and Figure 2bis a flow chart of an embodiment of a method for evaluating an updated analysis program in a monitoring system including a plurality of monitoring devices.

[0061] Figure 3 An exemplary system according to an embodiment of the present disclosure is shown.

[0062] Figure 4 An exemplary system according to another embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0063] The present invention will now be described more fully with reference to the accompanying drawings, in which presently preferred embodiments of the invention are shown. However, the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided for thoroughness and completeness and to convey to the skilled artisan the scope of the invention as defined in the appended claims.

[0064] Figure 1 An embodiment of the present invention is illustrated, including an evaluation system 200 that can be used to test and validate updated analysis programs, such as computer-assisted methods for identifying objects or actions in the same or similar physical locations monitored by a monitoring system 10. The updated analysis program is evaluated on selected monitoring devices 12 identified as having available processing resources. If the evaluation of the updated analysis program is successful, the update can be rolled out to the other monitoring devices 14, 16 of the system 10.

[0065] It will be understood that the monitoring system 10 may include or may be, for example, a radar system, an audio system, or a camera system. Figure 1 In the following description of , an exemplary embodiment is disclosed that includes a camera system 10. The camera system 10 includes a plurality of cameras 12, 14, 16. In this example, all three cameras 12, 14, 16 can be considered to monitor a similar environment or scene. The first camera 12 and the second camera 14 monitor the same three-dimensional space 20 or physical location from slightly different angles and positions, with at least partially overlapping fields of view FOV. Therefore, the respective scenes captured by the first camera 12 and the second camera 14 may be slightly different in terms of perspective, but still be considered similar in terms of common descriptors (i.e., features). The third camera 16 monitors a different three-dimensional space 22, but assuming that the two monitored physical locations 20, 22 belong to the same classification and / or share a common set of features, the resulting scene (or image) captured by the third camera 16 can be considered similar to the scenes captured by the first camera 12 and the second camera 14.

[0066] However, it will be appreciated that the first camera 12 in an alternative configuration may be arranged to monitor an environment different from that monitored by the second camera 14. In such a configuration, the first camera 12 may be employed primarily for its available processing resources, which may be used to execute the updated analysis routine.

[0067] The evaluation system 200 includes a circuit 210 configured to perform the functions of the evaluation system 200, i.e., to be combined with Figure 2a and Figure 2b The evaluation system 200 may include or be communicatively connected to a non-transitory computer-readable medium or memory 217 storing computer code instructions for executing the above-described method steps. In some embodiments, the evaluation system 200 may be distributed among one or more of the monitoring devices 12, 14, 16, or structurally separate from the monitoring system 10. In the latter case, the evaluation system 200 may be located remotely from the monitoring system 10 and may therefore be referred to as a central evaluation system 200, including a central processing device or central circuit.

[0068] Figure 2a to Figure 2b A flow chart showing an embodiment of a method 100 for evaluating an updated analysis program as described above, i.e., in a monitoring or surveillance system comprising a plurality of surveillance devices, at least some of which are arranged to image similar scenes. In the following, the surveillance system will be exemplified by a camera system comprising at least a first camera 12 and a second camera 14, as in Figure 1 As shown in the picture.

[0069] Method 100 includes an act of identifying S110 available processing resources in the camera system, for example by monitoring actual processor utilization or by estimating processor utilization based on historical data, with the goal of identifying and selecting a first camera suitable for executing the updated analysis program. Available processing resources can also be identified as a period of time during which the importance of monitoring a particular scene is low. As a specific illustrative example, a scene including a railway platform can be considered. In this case, the less important time period can be represented by the period after a train leaves the platform until the next train arrives. During this period, the risk of an accident may be determined to be relatively low, and therefore the importance of monitoring the scene may be low (from a safety perspective).

[0070] Once the available processing resources are identified S110, a first camera having available processing resources is selected S120. Preferably, a camera having sufficient available or excess resources to execute the updated analysis program may be selected S120 as the first camera.

[0071] Method 100 also includes selecting a second camera from the plurality of cameras (S130). For example, the second camera may be a camera that has been determined to require an upgrade to its current analysis program. For example, the second camera may exhibit impaired performance compared to other cameras in the system. The second camera may be used to capture (S140) an image, which may be sent (S155) to the first camera. In this example, the current analysis program may be executed (S150) on the image by the second camera. However, it should be understood that in some examples, the current analysis program may also be executed by another entity, such as a central processing unit or the first camera.

[0072] The result of the current analysis procedure may be referred to as a first result, which may include, for example, information about objects or actions recognized in the image.

[0073] Once the image is received at the first camera, the image is analyzed by the updated analysis program, and the updated analysis program is executed S160 to generate a second result. Similar to the above, the second result may include information about the object or action recognized in the image.

[0074] The first result is used to calculate S170 a first performance value indicating the performance of the second camera when executing the current analysis program. For example, the first performance value can be considered a quality metric that quantifies the accuracy or reliability of the current analysis program, or a metric that indicates processor utilization or time consumption for executing the current analysis program.

[0075] Similarly, the second result is used to calculate S180 a second performance value, which for the updated analysis program may indicate a similar metric as the current analysis program described above. The calculations S170, S180 may be performed at the first camera or the second camera, or at a centrally located processing device such as a central processing unit. In this example, the first result is sent S157 to the first camera, which may calculate S170 the first performance value. In addition, as in Figure 2b As illustrated in , the second result may be sent S165 to a second camera where a second performance value may be calculated S180 .

[0076] Since both the current and updated analysis processes have been performed on the same image captured by the second camera, the updated analysis program is evaluated S190 by comparing the first performance value and the second performance value. For example, the evaluation may take into account the time taken to execute the current and updated analysis programs, respectively. In a further example, the performance value may indicate the power consumption used when executing the current and updated analysis programs, respectively. For example, the result of the evaluation may indicate that there is an improvement, or at least that the updated analysis program meets the specifications and requirements, thereby achieving its intended purpose. Preferably, the first camera and the second camera (and possibly other cameras, such as all cameras of the camera system) can be of the same or similar hardware type, i.e., include the same or similar type of hardware for executing the analysis program, so that the performance of the updated analysis program on the first camera is more representative of the second (and more) cameras of the camera system.

[0077] Thus, the method may include the further act of sending S195 the updated analysis program to the second camera, assuming that the evaluation indicates an improvement or that the updated analysis program meets the specifications and requirements associated with the analysis program. The requirements and specifications may be defined on a case-by-case basis, for example by an installer or operator based on other and overall requirements of the surveillance system.

[0078] Figure 3 2 is a schematic block diagram of an embodiment of an evaluation system 200 for evaluating an updated analysis program in a surveillance system, including, for example, a camera system, a radar system, a lidar system, or an audio system as described above. The evaluation system 200 includes circuitry 210 configured to perform the functions of the evaluation system 200, such as with respect to Figure 2a to Figure 2bThe circuit 210 may include a processor 215, such as a central processing unit (CPU), a microcontroller, or a microprocessor. The evaluation system 200 may also include a non-transitory computer-readable storage medium, such as a memory 217. The memory 217 may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, a random access memory (RAM), or other suitable devices. In a typical arrangement, the memory 217 may include a non-volatile memory for long-term data storage and a volatile memory used as a system memory for the circuit 217. The memory 217 may exchange data with the circuit 210 via a data bus. There may also be accompanying control lines and an address bus between the memory 217 and the circuit 210. The memory may include instructions 219 in the form of program code that are configured to perform the functions of the evaluation system 200 when executed by the processor 215. The functions of the evaluation system 200 can be, for example, to identify available processing resources in a monitoring system, such as the camera system 10, select a first monitoring device from a plurality of monitoring devices (for which available processing resources have been identified), such as the first camera 12, select a second monitoring device from the plurality of monitoring devices (such as the second camera 14) to acquire monitoring data (such as an image), use the second camera 14 to send the image to the first camera 12, perform a current analysis program on the image to produce a first result, use the first camera 12 to perform an updated analysis program on the image to produce a second result, calculate a first performance value based on the first result, calculate a second performance value based on the second result, and evaluate the updated analysis program based on a comparison between the first performance value and the second performance value.

[0079] In some examples, cameras 12, 14 of camera system 10 may constitute part of evaluation system 200. In other examples, cameras 12, 14 of camera system 10 may be structurally separate from evaluation system 200. In the latter case, evaluation system 200 may be located remotely from camera system 10. In further examples, evaluation system 200 or portions of evaluation system 200 may be distributed throughout camera system 10, such as in first camera 12, second camera 14, and / or in a remote location.

[0080] Figure 4 is a schematic block diagram of an embodiment of an evaluation system 200, which may be similarly configured in conjunction with the above Figure 3 The evaluation system discussed. Figure 4 As shown, the circuit 210 is configured to perform at least an identification function 220 , a selection function 230 , an image capture function 240 , an image transmission function 250 , an analysis program execution function 260 , a calculation function 270 , and an evaluation function 280 .

[0081] The analysis program execution function 260 can adopt a deep learning method, which can be understood as a type of machine learning that may involve a training model, commonly referred to as a deep learning model. A deep learning model can be based on a set of algorithms that aim to model abstractions in the data by using multiple processing layers. The processing layers can be composed of nonlinear transformations, and each processing layer can transform the data before passing the transformed data to a subsequent processing layer. The transformation of the data can be performed by the weights and biases of the processing layers. The processing layers can be fully connected. By way of example and not limitation, a deep learning model can include a neural network and a convolutional neural network. A convolutional neural network can be composed of a hierarchy of trainable filters, interwoven with nonlinearity and pooling. Convolutional neural networks can be used for large-scale object recognition tasks.

[0082] Deep learning models can be trained in either a supervised or unsupervised setting. In a supervised setting, deep learning models are trained using labeled datasets to accurately classify data or predict outcomes. When input data is fed into a deep learning model, the model adjusts its weights until the model achieves an appropriate fit as part of the cross-validation process. In an unsupervised setting, deep learning models are trained using unlabeled datasets. From this unlabeled dataset, deep learning models discover patterns that can be used to cluster the data in the dataset into groups with common attributes. Common clustering algorithms include hierarchical, k-means, and Gaussian mixture models. Therefore, deep learning models can be trained to learn representations of data.

[0083] The input to the deep learning model can therefore be monitoring or surveillance data, such as Figure 4 In this example, image data from a camera device of a camera system is indicated in .

[0084] There are many different deep learning networks suitable for performing analytics on surveillance data, and the detailed form of inputs and outputs, i.e., what format the input data requires and what format the output tensors have, can vary between these networks. The output from a deep learning model can be interpreted using thresholding techniques, where the tensor outputs from the deep learning model are interpreted by setting a confidence score threshold for each analytic result, e.g., for each classification or category. The threshold sets the minimum confidence score required to interpret the data as a specific analytic result, e.g., belonging to a specific object category in the case of an object detector.

[0085] After training a deep learning model, the analysis program can be considered updated, for example in response to access to new input data. The analysis program can further be considered updated by updating or modifying the deep learning framework used to design, train, and interact with the deep learning model.

[0086] Those skilled in the art will appreciate that the present invention is not limited to the foregoing embodiments. Rather, numerous modifications and variations are possible within the scope of the appended claims. By studying the drawings, this disclosure, and the appended claims, those skilled in the art will understand and implement such modifications and variations in practicing the claimed invention.

Claims

1. A method (100) for evaluating an updated version of an analysis program in a monitoring system (10) comprising a plurality of monitoring devices (12, 14, 16), wherein: At least some of the plurality of monitoring devices are arranged to monitor similar environments, the method being characterized by: identifying ( S110 ) available processing resources in the monitoring system; selecting ( S120 ) a first monitoring device from the plurality of monitoring devices, for which available processing resources have been identified; After selecting the first monitoring device, selecting (S130) a second monitoring device from the plurality of monitoring devices to monitor an environment similar to that of the first monitoring device; Acquiring (S140) monitoring data by the second monitoring device; executing ( S150 ) a current version of an analysis program on the monitoring data, thereby generating a first result; sending ( S155 ) the monitoring data acquired ( S140 ) by the second monitoring device to the first monitoring device; executing (S160) the updated version of the analysis program on the monitoring data in the first monitoring device, thereby generating a second result; calculating ( S170 ) a first performance value based on the first result, the first performance value indicating the performance of the second monitoring device when executing the current version of the analysis program; calculating ( S180 ) a second performance value based on the second result, the second performance value indicating the performance of the first monitoring device when executing the updated version of the analysis program; as well as The updated version of the analysis program is evaluated ( S190 ) based on a comparison between the first performance value and the second performance value.

2. The method according to claim 1, wherein The first performance value and the second performance value indicate time, used power resources, or used processing resources for executing the current version of the analysis program and the updated version of the analysis program, respectively.

3. The method according to claim 1, wherein The first performance value and the second performance value indicate the accuracy or reliability of the current version of the analysis program and the updated version of the analysis program, respectively.

4. The method according to claim 1 or 2, wherein: The second monitoring device executes the current version of the analysis program on the monitoring data and sends ( S157 ) the first result to the first monitoring device that calculates the first performance value.

5. The method according to claim 1 or 2, wherein: The first monitoring device sends ( S165 ) the second result to the second monitoring device, and wherein the second monitoring device calculates the first performance value and the second performance value.

6. The method according to claim 1 or 2, further comprising: In case the evaluation indicates an improvement, the updated version of the analysis program is transmitted (S195) to the second monitoring device.

7. The method according to claim 1 or 2, further comprising: In case the evaluation indicates an improvement, the updated version of the analysis program is transmitted (S195) to a third monitoring device of the plurality of monitoring devices, wherein the second monitoring device and the third monitoring device are arranged to monitor a similar environment.

8. The method according to claim 1 or 2, wherein: Selecting the first monitoring device includes identifying a monitoring device from the plurality of monitoring devices that has sufficient available processing resources to execute and evaluate the updated version of the analysis program.

9. The method according to claim 1 or 2, wherein: The first monitoring device includes first hardware for executing the updated version of the analysis program, wherein the second monitoring device includes second hardware for executing the current version of the analysis program, wherein each of the first hardware and the second hardware includes a hardware accelerator, a CPU or a GPU, and wherein the first hardware and the second hardware are of similar types.

10. The method according to claim 1 or 2, wherein: The updated version of the analysis program is executed using a deep learning model operating in a deep learning framework, and wherein at least one of the deep learning model and the deep learning framework is updated.

11. The method according to claim 1 or 2, wherein: The current version of the analysis program and the updated version of the analysis program include object recognition.

12. An evaluation system (200) for evaluating an updated version of an analysis program in a monitoring system (10) comprising a plurality of monitoring devices (12, 14, 16), wherein: At least some of the plurality of monitoring devices are arranged to monitor similar environments, the evaluation system being characterized in that the circuit (210) is configured to perform: an identification module configured to identify available processing resources in the monitoring system; a selection module configured to select a first monitoring device from the plurality of monitoring devices, for which available processing resources have been identified, and further configured to select a second monitoring device from the plurality of monitoring devices after selecting the first monitoring device, the second monitoring device monitoring an environment similar to that of the first monitoring device; a monitoring data acquisition module, configured to acquire monitoring data using the second monitoring device; a monitoring data sending module, configured to send the monitoring data acquired by the second monitoring device to the first monitoring device; an analysis program execution module configured to execute a current version of the analysis program on the monitoring data, thereby generating a first result, and further execute the updated version of the analysis program on the monitoring data using the first monitoring device, thereby generating a second result; a calculation module configured to calculate a first performance value indicating the performance of the second monitoring device when executing the current version of the analysis program based on the first result, and further calculate a second performance value indicating the performance of the first monitoring device when executing the updated version of the analysis program based on the second result; as well as An evaluation module is configured to evaluate the updated version of the analysis program based on a comparison between the first performance value and the second performance value.

13. The evaluation system according to claim 12, wherein: The surveillance system includes a camera system, the first surveillance device includes a first camera, and the second surveillance device includes a second camera.

14. The evaluation system according to claim 12 or 13, wherein: The monitoring system includes at least one of the following: A radar system, wherein the first monitoring device comprises a first radar device and the second monitoring device comprises a second radar device, and An audio system wherein the first monitoring device comprises a first audio device and the second monitoring device comprises a second audio device.

15. A non-transitory computer-readable medium having computer code instructions stored thereon, the computer code instructions being adapted to perform the method of claim 1 when executed by a device having processing capabilities.

Images