METHOD FOR ANALYZING VIDEO CONTENT AND DEVICE EXECUTING THE METHOD.

Abstract

The invention relates to a method for analyzing video content comprising determining (S1) a list of objects and / or events to be detected in one or more video clips, determining (S2) a list of resources necessary for detecting said objects and / or events, decomposing (S3) the necessary resources into a plurality of execution modules of a processing algorithm, and automatically calculating (S4) an optimized information traversal between said execution modules, so as to minimize the number of said execution modules required to detect elements from said list of objects and / or events. The invention also relates to a video content analysis device configured to execute the method. Figure to be published with the abstract: Fig. 2

Description

Title of the invention: METHOD FOR ANALYZING VIDEO CONTENT AND DEVICE CARRYING OUT THE METHOD. Technical field

[0001] The present invention relates to the field of video stream analysis. The invention relates more particularly to the allocation of hardware and / or software resources necessary for operations of object detection or event detection in video content, from video streams. STATE OF PRIOR ART

[0002] Video stream analysis systems make it possible to carry out automatic detections of objects or events, in real time or in a delayed manner, from video streams generated by image capture devices such as, for example, video surveillance cameras. Such systems are particularly common and useful in urban areas to ensure, for example, the proper use of road traffic infrastructure, the proper application of vehicle parking rules, the safety of property and people, etc. The applications of video surveillance are numerous and varied, and the above examples are not limiting.Thus, video surveillance systems are commonly used for parking management, to encourage compliance with traffic lights and the correct direction of travel, to secure pedestrian walkways or railways, but also to ensure the safety of people in places where there is a high level of traffic, such as train stations, airports, or shopping centers. Such systems are also appreciated for their contribution to traffic management, crowd management, and more broadly for their ability to help predict and plan urban developments or urban infrastructure developments. The success of these automated image capture and analysis systems, combined with the ease of use offered by the distribution of "turnkey" applications, is such that the need for the number of image capture devices is constantly increasing.The requirements for hardware and software resources implemented for the automated analysis of produced video streams have increased accordingly. These systems are complex and expensive and there is a need to optimize the allocation of hardware and / or software resources required for analysis operations for predetermined or evolving needs. The situation can therefore be improved. Statement of the invention

[0003] The aim of the invention is to propose a method for allocating useful resources to video content analysis operations aimed at ensuring that the detections to be carried out can be achieved, or in other words that no saturation in terms of resource usage can harm good detection of an object or event from one or more video contents.

[0004] To this end, a method for analyzing video content is proposed, executed in a video content analysis device, comprising:

[0005] - determine a list of objects and / or events to be detected in one or more video content,

[0006] - determine a list of resources necessary for the detection of objects and / or events to detect,

[0007] - decompose the resources into a plurality of modules for executing an algorithm processing, each of the modules being configured to process all or part of one or more images extracted from one of the video contents, or information representative of a video content obtained from one or more of the images and at least one of the execution modules being configured to deliver information representative of a result of a detection to be carried out, the method further comprising,

[0008] - automatically determine an optimized information path between modules execution so as to minimize the number of execution modules required to operate element detections among the list of objects and / or events to be detected.

[0009] Advantageously, the use of hardware and / or software resources is optimized to enable a large number of detections to be carried out while limiting as much as possible the risk of not detecting an object or event sought in one or more video contents.

[0010] According to one embodiment of the invention, the steps of the method for determining a list of necessary resources and for breaking down the resources into a plurality of execution modules of a processing algorithm comprise determining or considering an index representative of a processing capacity of each of the execution modules.

[0011] Thus, the allocation (or in other words the distribution) to execution modules of a processing algorithm of the global resources to be used to carry out the programmed analyses and detections is facilitated.

[0012] According to one embodiment, the method further comprises a step of determining a rate of resources used with regard to all of the available resources of the video content analysis device and at least one step of executing a video content analysis process is conditioned on the fact that the rate of resources used exceeds or does not exceed a predetermined threshold value.

[0013] According to one embodiment of the invention, the predetermined threshold value is between 80 and 90% of the total resources available in the video content analysis device.

[0014] According to one embodiment of the invention, the optimized data path is implemented using asynchronous messaging functions for distributed or concurrent applications, preferably according to a protocol of a so-called ZeroMQ type or a protocol of an MQTT type (defined according to the ISO / IEC 20922:2016 standard or one of its evolutions).

[0015] The invention also relates to a video content analysis device comprising a set of configured electronic circuits including:

[0016] - a module configured to determine a list of objects and / or events to detect in one or more video contents,

[0017] - a module configured to determine a list of resources necessary for the detection of objects and / or events to be detected,

[0018] - a module configured to decompose the resources into a plurality of modules execution of a processing algorithm, each of the modules being configured to process all or part of one or more images extracted from one of the video contents, or information representative of a video content obtained from one or more of the images and at least one of the execution modules being configured to deliver information representative of a result of a detection to be carried out, the video content analysis device further comprising:

[0019] - a module configured to automatically determine an optimized route of information between the execution modules of an algorithm, so as to minimize the number of execution modules required to operate detections of elements among the list of objects and / or events to be detected.

[0020] The invention also relates to a video surveillance system comprising one or more video content analysis devices such as the video content analysis device previously described.

[0021] Finally, the invention relates to a computer program product comprising program code instructions for executing the steps of the aforementioned method, when the program is executed by a processor, as well as a storage device comprising such a computer program product. Brief description of the drawings

[0022] The characteristics of the invention mentioned above, as well as others, will appear more clearly on reading the following description of at least one exemplary embodiment, said description being made in relation to the attached drawings, among which:

[0023] [Fig.l] schematically illustrates a video surveillance system comprising a video content analysis device executing a video content analysis method according to one embodiment;

[0024] [Fig.2] is a flowchart representing a method of analyzing video content according to one embodiment, executed in the video surveillance system already illustrated in [Fig. 1];

[0025] [Fig.3] is a diagram representing the architecture of the video content analysis device shown in [Fig.l] and configured to execute a method described in [Fig.2].

[0026] DETAILED DESCRIPTION OF EMBODIMENTS

[0027] [Fig.l] schematically illustrates a video surveillance system 10 comprising a device 100 for analyzing video content. The video content analysis device 100 is configured to perform a detection of predefined objects and / or events from video content delivered by image capture devices 102, 104, 106 and 108. The image capture devices 102, 104, 106 and 108 are respectively connected to the video content analysis device 100 via communication buses 103, 105, 107 and 109. According to one embodiment, the image capture devices 102, 104, 106 and 108 are cameras called “IP cameras” configured to be controlled remotely and to each deliver one or more video contents from a shooting device in the form of a digital video stream.The term "digital video stream" here designates a succession of identifiable information having a link between them and being representative of one or more video contents (for example one or more films or video sequences) and whose processing (decoding) allows a restitution of one or more video contents. More broadly, a video stream can also comprise an audio stream coding a sound environment around a camera. In this case, the video stream is more precisely an audio / video or audiovisual stream. Thus, each of the communication buses 103, 105, 107 and 109 comprises communication lines useful for transporting video streams and communication lines useful for configuration operations of the image capture device to which it is connected.The communication lines useful for transporting video streams can also be used to transport data related to the video streams, such as, for example, camera identifiers, camera movement control data, video stream timestamp data. The camera movement control data makes it possible, for example, to control a rotating camera to modify or adjust its field of view. According to one embodiment, the communication lines provided for transporting video content from an image capture device are also provided for configuring the image capture devices. According to this embodiment, a communication bus thus produced is bidirectional and a data transfer sequencing is carried out to interleave (or in other words to alternate) operations for configuring an image capture device and . operations for transporting video content between the image capture device concerned and the video content analysis device 100.

[0028] According to one embodiment, the video streams transporting the video content each use an application-level communication protocol of the RTSP type (from the English acronym for “Real-Time Streaming Protocol” and which means “real-time streaming protocol”).

[0029] The terms “video content” here designate all the visible and identifiable elements in a video sequence generated by an image capture device among the image capture devices 102, 104, 106 and 108. According to the example described, a video sequence corresponds to a shooting plane or, for example, to a shooting plane for a determined magnification factor (zoom level). Indeed, some of the processing operations applied to video content can interpret successive shooting planes when a rapid modification of the captured image occurs (for example a modification of a zoom level, or a modification by isometry or by homothety, due to rotational or translational movements of a capture device).In other words, a video sequence is a series of images corresponding to the field of view of an image capture device, i.e. the total area of ​​space perceived by the camera, possibly temporally delimited by a rapid modification of shooting characteristics, and therefore of the image as captured. Thus, a video sequence comprises only a single shot if an image capture device generating it is not deactivated, unless this device is possibly configured to carry out a rapid modification of shooting characteristics, as previously described, which then constitutes a transition between the end of a video sequence and the beginning of another video sequence, according to the different processing algorithms applied to the sequence.

[0030] The video surveillance system 10 further comprises, connected to the video content analysis device 100, an analysis control device 110 and an analysis results restitution device 112. These two devices 110 and 112 are respectively connected to the video content analysis device 100 via bidirectional communication buses 111 and 113. The video surveillance system 10 further comprises an analysis results storage device 114 which allows the recording of video content analysis results carried out by the video content analysis device 100.The analysis results recorded in the storage device 114 are, for example, notifications relating to detections of an object or an event, in the form of detection logs, images illustrating detections made, or even combinations of notifications and one or more images illustrating the detections of which notification is made. The storage device 114 is connected to the video content analysis device 100 via a bus. communication 115. According to one embodiment, the communication buses 111 and 113 are bidirectional communication buses and the communication bus 115 is a unidirectional or bidirectional communication bus. Each of the communication buses 103, 105, 107, 109, 111, 113 and 115 may be of the wired type or of the wireless type depending on the configuration, and more particularly depending on the topology of the video surveillance system 10, or, in other words, depending on the geographical distance between the different elements described and which make up the video surveillance system 10.

[0031] The analysis control device 110 is provided and configured to program detections to be carried out by the video surveillance system 10 in the video contents delivered to the video content analysis device 100 via the communication buses 103, 105, 107 and 109, from information entered by a user, who is also commonly called the operator of the system. Thus, a user of the system can configure the video content analysis device 100 to program a detection of one or more objects and / or one or more events in one or more of the different video contents transmitted by the image capture devices 102, 104, 106 and 108 connected to the video content analysis device 100. For example, an operator can program a detection of a pedestrian in video content transmitted by the image capture device 102.Similarly, an operator of the analysis control device 110 can program a detection of an isolated piece of luggage in this same video content. These detections then correspond, according to these examples, to detections of objects. Similarly again, it is possible to program a detection of an event related to one or more objects. It should be noted that a detection of an event here includes, by definition, the consideration of a temporality in addition to one or more facts (or states) relating to one or more objects. For example, the video surveillance system 10 is configured to allow the detection of luggage abandoned by a person, which constitutes an event related to objects that are the person and the luggage. This implies, for example, a detection of a person (object) and a piece of luggage (object), initially traveling the same path in time and space, then being moved away from each other.An event that characterizes the abandonment of baggage by a person can be broken down into an event of a person carrying baggage, followed by an event corresponding to the fact that this person physically separates from this baggage, then an event corresponding to a failure to recover this baggage by a person for a predetermined period.

[0032] The video content analysis control device 110 makes it possible to program detections of objects or events among a predefined set of objects or events from one or more identifiable video contents.

[0033] More generally, the video surveillance system 10 makes it possible, thanks to the analysis control device 110, to detect a predefined object or a predefined event in a video content. A detection of an object is then, in other words, a detection of an element likely to be identified as such (as this object) thanks to the shapes and visual characteristics (texture, color, etc.) that it presents to an image capture device (a camera, for example) and therefore to a shape that it presents in a video content. By way of non-limiting examples, such an object can be, for example, a face, a pedestrian, a car, a bus, a road sign, luggage, a hat, a pair of glasses, a bottle of milk, etc. A detection of an event is a detection of a fact relating to one or more objects in a video content, and which in fact includes a temporality.For example, an event thus defined may be an appearance of an object in video content, a movement of an object in video content, a disappearance of an object in video content, a relative movement of an object in relation to another object, in video content (according to the example of luggage carried by a person, for example), a variation in movement of an object in relation to another object in video content (approaching, moving away, etc.).Thus, and in other words, non-limiting examples of events that may be searched for and detected in video content are: an appearance of luggage, a disappearance of luggage, an appearance of a pedestrian, a disappearance of a pedestrian, a carrying of luggage by a pedestrian, a deposit of luggage by a pedestrian, a grasping of luggage by a pedestrian, an immobility of luggage for a predetermined time, but also, according to other non-limiting examples, a car crossing a red light, a vehicle entering a one-way street, against the flow of traffic, a collision between two vehicles, a football crossing a touchline of a football stadium, a candidate talking with a third person during a school exam monitored by video surveillance means, etc.

[0034] Advantageously, the operations of detecting objects in video content can be carried out using so-called computer vision techniques based on the use of neural networks trained from image samples during a learning phase.

[0035] According to one embodiment, an object detection module comprises a software or hardware implementation of a deep artificial neural network or DCNN (Deep Convolutional Neural Network). Such a DCNN module may consist of a set of numerous artificial neurons, of the convolutional type or of the perceptron type, and organized by successive layers connected to each other. Such a DCNN module is classically inspired by a simplistic model of the functioning of a human brain where numerous biological neurons are connected to each other by axons.

[0036] For example, a module called YOLOv4 (from the English acronym “You Only Look Once version 4”) is a DCNN type module which allows object detection in images, and which is called “one-stage”, that is to say whose architecture is composed of a single module of combined proposals of rectangles framing objects (“bounding boxes”) and classes of objects in the image. In addition to the artificial neurons previously described, the YOLOv4 module uses functions known to those skilled in the art such as for example batch normalization, dropblock regularization, weighted residual connections between layers or a non-maximal suppression step which eliminates redundant proposals of detected objects.

[0037] According to one embodiment, the object detection module has the possibility of predicting a list of objects present in one or more images of one or more video contents by providing, for each object, a rectangle framing the object in the form of coordinates of points defining the rectangle in the image, the type or class of the object from a predefined list of classes defined during a learning phase, and a detection score representing a degree of confidence in the detection thus carried out.

[0038] The analysis results restitution device 112 is a device provided and configured to operate a restitution of the results of one or more analyses carried out by the video surveillance system 10 by means of the video content analysis device 100. Thus, the analysis results restitution device 112 is provided to notify an operator of the detections of objects and / or events corresponding to the objects and / or events searched for in the video content. According to one embodiment, the analysis results restitution device 112 displays an alert message for each detection carried out and which corresponds to a detection of an object and / or an event as programmed via the analysis control device 110. According to one embodiment, the restitution of an alert message can be visual, audible, or visual and audible and allows an operator to obtain information related to a detection carried out.For example, a notification of a detection via the analysis results restitution device 112 may be a line of succinct information, displayed on a screen, and comprising a summary of detailed information which can be accessed by selecting the line. The succinct summary may include, by way of example, the type of detection carried out (object or event), a title of the detection (for example, crossing a red light on an urban traffic lane), a date and a time.

[0039] According to one embodiment, the video content analysis device 100 is configured to determine a list of objects and / or events to be detected in one or more video contents from information entered by an operator, via the analysis control device 110. For example, the video content analysis device 100 is configured to detect an event of the “abandoned baggage” type in a video content transmitted from the image capture device 102, which image capture device 102 is positioned in the entrance hall of an airport. According to another example, the video content analysis device is configured to detect an event of the “ball out of touch” type in a video content transmitted from the image capture device 103, which image capture device 103 is positioned in a touchline of a football stadium.The video content analysis device 100 is further configured to determine, from the list of objects and / or events to be detected, entered or updated by an operator, a list of resources necessary for the detection of these objects and / or events to be detected, then to break down the resources into a plurality of modules for executing a processing algorithm, considering that each of the modules for executing a processing algorithm is configured to process all or part of one or more images extracted from one of the video contents, or is configured to process information representative of a video content obtained from one or more of the images. The modules for executing a processing algorithm are also called “processors”. At least one of the execution modules is configured to deliver information representative of a result of a detection to be carried out.In other words, this amounts to saying that the system is configured to use resources in the form of elementary image processing modules which are advantageously dynamically allocated to enable all of the successive image processing operations required to analyze the video content and detect the different objects and / or events to be detected. For example, an elementary module for executing an image processing algorithm may be a module for detecting the presence of an object in a given image, extracted from video content or pointed to in video content. More precisely, an example of such an elementary module for executing an algorithm may be a module for detecting the presence of an object in an image identified in given video content.Such a module can be defined with input data that are, for example, an identifier of a time-stamped image in a video content (or a link to it), an identifier of an object to be searched for in this image and output data of this module are then an indicator of presence of the searched object, the coordinates of a precise point (reference point) of the object in the image and dimensions of a frame delimiting the object around the precise point of the object. A reference point is for example the center of . the object or the center of a median or the lower-left point of a frame bordering the object, etc.

[0040] By way of example, an elementary module for executing an object detection processing can be configured and used to detect the presence of an object of the “luggage” type in an image extracted from video content coming from the image capture device 102, the image being timestamped and captured at a precise date and time (for example July 12, 2021 at 12:16:34 p.m.), with a timestamping precision of the order of a second, for example.

[0041] According to a similar operation, this same object detection module can be used to detect the presence of a “pedestrian” type object in the same image or in another image, or even in another video content, at a time close to or far from the search for luggage. Furthermore, an elementary module for executing an algorithm can be defined for the detection of an event linked to two objects in the same image. For example, an elementary module for executing an event detection algorithm can be configured to detect an identical or almost identical movement of two objects in the same image, the presence of which has already been detected by a previous processing (and by a module for executing a different algorithm).Thus, such an elementary processing module can have as input data a link to an image, respective coordinates of reference points of each of the two previously detected objects, and a predefined time interval (for example, a start time and an analysis duration), and an output data of this elementary module would then be an indicator of whether or not the movements of the two objects match.Advantageously, the video content analysis device comprises a library of algorithm execution modules (sometimes also called processors) for processing comprising numerous elementary algorithm execution modules using as input data all or part of an image extracted from video content or using as input data information representative of one or more elements contained in an image (for example, the coordinates of a reference point of a detected object, in an image, and the dimensions of a frame delimiting this object, relative to the reference point).

[0042] Advantageously, the video content analysis device 100 is configured to automatically calculate or determine, from a list of detections to be carried out and a list of elementary modules for executing an algorithm, an optimized path of information between the execution modules, so as to minimize the number of execution modules required to carry out the detections of programmed elements among the list of objects and / or events to be detected.

[0043] An execution module of a processing algorithm can be implemented under software form or in hardware form. When implemented in software form, an execution module of an implemented processing algorithm may be a new instance of such an execution module, executed in a RAM of the video content analysis device 100, under the control of a control unit of the video content analysis device 100. When implemented in hardware form, the execution module may be an nth instance of a determined module, implemented in a finite number of instances in a logic circuit, for example, of the video content analysis device 100, and configured to carry out a detection.Obviously, the resources are available in a finite number, whether due to the hardware architecture and / or the size of the RAM of the device 100, and it is therefore appropriate to optimize their use so as not to risk a lack of detection of an object or an event due to the temporary unavailability of a resource or even to be able to increase the number of detections to be carried out with regard to a given architecture (and therefore a given number of resources).This is why an automatic calculation, from a list of detections to be carried out and a list of elementary modules for executing an algorithm, of an optimized path of information between the execution modules, so as to minimize the number of execution modules required to operate detections of elements among the list of objects and / or events to be detected, makes it possible to optimize the allocation of resources and to limit as much as possible a risk of absence of detection in the presence of an object or an event to be detected in video content.

[0044] According to one embodiment, the determination of a list of necessary resources and the decomposition into a plurality of modules for executing a processing algorithm uses a system of indices representative of a processing capacity for each of the modules for executing an algorithm as defined in the library of elementary modules for executing a processing algorithm of the video content analysis device 100. Thus, a coefficient Xi expressed in processing capacity units is associated with each elementary module (indexed i) for executing an algorithm as defined in the library of elementary modules for executing an algorithm of the video content analysis device 100. A coefficient Xi can also be expressed in capacity units per unit of time.It should be noted that resource requirements are not linear, i.e., the amount of resources required to detect ten events in a video content is not equal to ten times the amount of resources required for a single detection. In other words, the relationship between the resources required to detect a number of objects or events in one or more video contents and this number of objects or events is not necessarily linear.

[0045] Advantageously, the video content analysis device 100 comprises a control module called “supervisor” making it possible to define in a recurring manner a rate of resources used with regard to all the available resources of the video content analysis device 100. It is then possible to carry out a conditioned execution of a video content analysis process depending on whether or not said rate exceeds a predetermined threshold value. For example, the predetermined threshold value can be between 60% and 95% of the total resources available in the analysis device 100, preferably between 80 and 90% of these resources.In the case where the overall resources are insufficient to carry out all of the detections programmed by an operator, depending on other parameters of the video surveillance system, such as, for example, the number of image capture devices or the resolution of the images of the video content, the video content analysis device 100 is configured to issue an alert to the operator, via the analysis control device 110 or via the analysis results restitution device 112. Advantageously, a module for executing a processing algorithm can be put into operation conditionally according to a level of use of the available resources, also called the “saturation level” of the resources. Thus, a processing carried out by a module for executing an algorithm can be executed conditionally according to the level of use of the resources of the analysis device 100.

[0046] According to one embodiment, the optimized path of the data between the different analysis modules is implemented using asynchronous messaging functions for distributed or concurrent applications, preferably according to a protocol of a type called ZeroMQ also commonly called ZMQ, a protocol of an MQTT type or another equivalent protocol. For this purpose, data transfers (also called here more broadly information) are carried out according to a ZMQ, MQTT protocol, or according to another equivalent protocol between output registers of the execution modules of a processing algorithm and data registers of execution modules of a processing algorithm, as soon as necessary, and according to, for example, a so-called "round robin" algorithm.

[0047] According to one embodiment, a characteristic is defined for each module of the library of elementary modules, so as to be able to determine whether a module is of a so-called “stateful” type or of a so-called “stateless” type.

[0048] A module defined as being “stateful” may require the use of information originating from or representative of prior processing, carried out by this same module, on all or part of one or more video contents or information representative of a video content. Such a module comprises an internal memory capable of recording information representative of a state of the module at a given time. This memory may, for example, be used to recreate an instance of this module during a resumption of processing following an error (sometimes also called “error recovery”). Such an error may be linked to an accidental unavailability of a hardware or software resource, for example following a breakdown, a “bug”, or an interruption of a network connection. For example, a module tracking a “car” object in several successive frames of video content may require previous positions of several “car” objects in previous frames from the same video content to then associate positions of “car” objects in a new frame with “car” objects tracked by the system in previously processed frames.

[0049] Conversely, a module defined as being of the “stateless” type is based only on input data presented to it and which are, for example, related to all or part of one or more video contents or information representative of a video content. For example, a module performing a detection of “car” type objects in an image may only need video content corresponding to this image but not video content corresponding to previous images.

[0050] According to one embodiment, an optimized path of information between the different execution modules is calculated by the control module, called “supervisor” so as to minimize the number of execution modules required to operate detections of elements among a list of objects and / or events to be detected on all or part of one or more video contents or information representative of a video content.

[0051] According to this embodiment, the control module called supervisor determines, to do this, a series of image processing operations to be carried out and respective configuration parameters to operate detections of elements among the list of objects and / or events to be detected.

[0052] The supervisor also determines a sequence of module types, each identified as being “stateful” or “stateless” and with which is associated a coefficient Xi as previously described and expressing a processing capacity associated with this module, necessary to carry out image processing operations.

[0053] The supervisor also defines the current use of the processing capacities of all the modules present in the video content analysis device 100.

[0054] The supervisor can then determine the list of execution modules identified as being “in existing state” and which have sufficient processing capacity available to carry out all the processing of a given type on one or more video contents or on information representative of a video content. The supervisor then selects, for each video content to be processed or each group of information representative of a video content to be processed, the execution module which performs processing from a list of available execution modules using a so-called “round-robin” algorithm. Note that video content that has been broken down into multiple sub-streams and processed in multiple modules during previous processing must be merged into a single stream and reordered to be processed by a module defined as “stateful”.

[0055] A distribution of detection tasks according to the round-robin algorithm amounts to distributing a first task (detection) to be processed to a first execution module in a list of available execution modules, then a second task to be processed to a second execution module in this list, and so on. The supervisor manages one or more dynamically updated allocation tables in a dedicated working memory, which is preferably internal or fast to access. The allocation tables make it possible to establish the link between the tasks to be operated and the execution modules responsible for carrying them out.

[0056] In particular, when new detection tasks are to be carried out, the allocation table(s) are updated accordingly.

[0057] In a similar manner, the supervisor determines a list of existing execution modules defined as being “stateless” and which can perform all the processing of a given type applied to one or more video contents or to information representative of a video content. The supervisor distributes the incoming flows, for a type of module, between each execution module in an equitable manner and in such a way that the use of the processing capacity is best distributed between the modules of this type, so that each module has the least processing to perform. A video content can be divided into several flows, distributed between several processing modules, the operations to be performed in an execution module defined as being “stateless” being temporally independent.

[0058] In a software embodiment, if the list of available execution modules defined as being “stateful” does not contain enough modules to operate element detections from a list of objects and / or events to be detected in all or part of one or more video contents or in information representative of a video content, then the supervisor instantiates one or more new modules within the limit of the total resources of the video content analysis device 100. The supervisor then adds this new module (this new module instance) to the list of available modules to then determine, for each operation, the execution module(s) selected by the round-robin algorithm to process one or more video contents or information representative of such content.

[0059] Similarly, if the list of available execution modules defined as “stateless” does not contain enough processing modules to perform element detections among a list of objects and / or events to be detected in all or part of one or more video contents or in information representative of a video content, then the supervisor instantiates one or more new modules within the limit of the total resources of the video content analysis device 100.

[0060] For example, for the detection of an event of the “bicycle parking” type, video content can be analyzed by a sequence of execution modules comprising one or more so-called “stateless” modules for detecting a “bicycle” object in an image, one or more so-called “stateful” modules for tracking a “bicycle” object in a series of consecutive images of video content and one or more so-called “stateful” modules for parking rules engine. In this example, the video content can be distributed between different so-called “stateless” modules for detecting a “bicycle” object to speed up processing and will be gathered and ordered in a single stream to then be processed in a so-called “stateful” module for tracking a bicycle and in a so-called “stateful” module for parking rules engine.

[0061] A rules engine is defined as a set of so-called "business" rules, i.e. defined at an application level. For example, a rules engine may define that a traffic incident (for example, an infraction) occurs if a vehicle moves in a direction opposite to the only authorized direction of travel on a traffic lane intended for this vehicle, or that a motor vehicle moves on a lane reserved for the circulation of non-motorized vehicles.

[0062] According to one embodiment, the determination of a list of resources necessary for given processing operations, and the decomposition into a plurality of resources necessary for the execution of a processing algorithm, uses a system of indices representative of a target processing capacity, less than or equal to the processing capacity expressed by the coefficient Xi, for each of the execution modules of an algorithm as defined in the library of elementary modules for executing a processing algorithm. Thus, a coefficient Yi expressed in units of target processing capacity is associated with each elementary module (indexed i) for executing an algorithm as defined in the library of elementary modules for executing an algorithm of the video content analysis device 100.The supervisor, when modifying the list of objects and / or events to be detected, or at regular intervals, determines the actual (current) use of the processing capacities of each execution module, and then instantiates one or more new modules within the limit of the total resources of the video content analysis device 100, if the consumption of a module reaches the coefficient Yi. This mechanism advantageously makes it possible to anticipate the needs of the video content analysis device 100 to reduce the risk of temporary interruption of processing of video content or processing of information representative of video content, the time of . the instantiation of new processing modules, in the event of an increase in processing capacity requirements. Indeed, the implementation of a new instance of an execution module can, depending on the type of module, require a delay of several seconds.

[0063] [Fig. 2] is a flowchart illustrating a method for analyzing video content executed in a video content analysis device according to one embodiment. The method comprises an initial step S0 at the end of which all of the elements of the video surveillance system 10 are initialized and are able to operate nominally. In particular, at the end of step S0, an operator of the video surveillance system 10 can configure the system by means of at least the analysis control device 110 by choosing one or more objects and / or one or more events to be detected, from a set of predefined objects or events, in the video content carried by the video streams coming from the image capture devices 102, 104, 106 and 108, so as to determine a list of objects and / or events to be detected via the video surveillance system 10.The term "list of objects and / or events" is to be interpreted broadly here, and such a list of objects and / or events may comprise a single element, of object type, or even a single element of event type, in the application sense of the term "event", since it has been explained previously that an event relates to at least one object considered in relation to time. Thus, for example, an operator may select during a step SI, using, for example, a user interface made available by the analysis control device 110, one or more objects to be detected, and / or one or more events to be detected, by indicating for each of the objects and for each of the events, where applicable, in which of the video contents the object or event must be detected.For example, an operator can program a detection of a crossing of a red traffic light, by a land vehicle, in video content coming from the image capture device 106, during the night, or even whatever the time of day. According to another example, an operator can program the detection of parking in a reserved parking space, by an unauthorized vehicle, in video content delivered from the image capture device 106. According to yet another example, an operator can program a detection of a marauder near a premises containing sensitive goods. Thus, at the end of step SI, a list of objects and / or events to be detected has been determined from all or part of the video content.This determination of a list can be either a creation of a list, starting from an empty list, or an updating of a list following an addition or following a removal from the existing list of elements to be detected. During a step S2, the video content device 100 determines, from the criteria and / or from . known parameters, a number of resources necessary to carry out the detections to be carried out, programmed during the previous step SI. The device 100 considers in particular the processing capacity of each of the elementary processing algorithm execution modules with regard to the characteristics of the transmitted video streams and the video contents which are contained in the video streams. For example, a detection module configured for the detection of an object in a video content comprising 30 images per second can carry out an object detection on n video contents in a given time interval, taking into account its processing capacity (n being a positive integer). The required execution modules are selected from a library of execution modules, also sometimes called “processor” or “processing processor”.Processing characteristics specific to each of the elementary execution modules are known so as to be able to define and consider a processing capacity index X for each of the execution modules. For example, a processing capacity index Xi can be determined for an elementary execution module of a processing algorithm operating on video content comprising 30 images per second and another processing capacity index X2 can be determined for the same elementary execution module operating on video content comprising 50 images per second.Thus, having regard to the processing capacity of each of the elementary modules for executing a processing algorithm, operating on video content whose characteristics are known and possibly also considering “real-time” constraints of execution needs, it is possible to determine a number of elementary resources, which further corresponds to hardware and / or software resources of the video content analysis device 100.

[0064] During a step S3, the necessary resources are broken down into elementary modules taking into account the detection operations to be carried out, for example, according to the number of objects to be detected and the number of events to be detected, as well as according to the nature of the objects and / or events to be detected. The processing execution modules can be used according to an assembly specific to the objects and / or events to be detected and to the elementary modules defined in the library. Thus, a detection of an event of the “fire in a vehicle” type can be broken down into a detection of a vehicle, a detection of a fire, and a similar movement of the “vehicle” and “fire” objects detected over time.An elementary object detection module can therefore be used to detect a vehicle in an image, when the input data of this module indicates that the object to be detected is a vehicle, then the same elementary module (hardware, or even the same software instance of this module) can be used to detect a fire object in the same image as the one where it detected a vehicle object, and finally. another elementary processing execution module intended to detect a similarity of movement can be used to detect that the vehicle object and the fire object are moving in the same way, that is to say in other words substantially at the same locations in space, and at the same time. According to the example described above, two elementary modules for executing a processing algorithm are to be used. One is an execution module for an algorithm for detecting an object, the other is an execution module for a proximity detection algorithm in time. Of course, many execution modules can be defined and combined, including hierarchically, so that an execution module can use a plurality of modules of a "lower" level combined with each other.

[0065] By proceeding in this way, execution modules of a processing algorithm use data, or more generally information, resulting from a processing result of one or more other execution modules having carried out prior processing.

[0066] Advantageously and according to one embodiment, the data, also called information here, are transmitted from and to the execution modules necessary to carry out programmed detections using an asynchronous messaging system for distributed or concurrent applications. According to one embodiment, the messaging system operates according to a protocol of the so-called ZeroMQ type or according to a protocol of the so-called MQTT type. Variants can of course be used by calling upon other similar or equivalent protocols.

[0067] Cleverly, during a step S4, an optimized path of the information is calculated (or determined) so as to optimize the number of resources required and to address only the smallest possible number of hardware resources or to execute only the smallest possible number of software instances of an execution module of a processing algorithm, where appropriate, to meet the needs of programmed detection of the objects and / or events identified and configured during step SI.

[0068] Advantageously, an algorithm of the so-called "round robin" type applied to the different execution modules required and to the different information that must pass between these modules makes it possible to optimize the number of resources and to avoid or limit as much as possible, consequently, the risk of not detecting a sought element in video content of a video stream.

[0069] [Fig. 3] schematically illustrates an example of the internal architecture of the video content analysis device 100. The architecture shown may also represent the internal architecture of the analysis control device 110, of the analysis results restitution device 112, of the analysis results storage device 114, or of any of the image capture devices (camera) 102, 104, 106 and 108.

[0070] According to the example of hardware architecture represented in [Fig. 3], the video content analysis device 100 then comprises, connected by a communication bus 1000: a processor or CPU (“Central Processing Unit” in English) 1001 comprising graphic processing units called GPU; a RAM (“Random Access Memory” in English) 1002; a ROM (“Read Only Memory” in English) 1003; a storage unit such as a hard disk (or a storage media reader, such as an SD (“Secure Digital” in English) card reader) 1004; at least one communication interface 1005 allowing the video content analysis device 100 to communicate with third-party devices such as the image capture devices 102, 104, 106 and 108, or the analysis control devices 110 or analysis results restitution devices 112.

[0071] The processor 1001 is capable of executing instructions loaded into the RAM 1002 from the ROM 1003, from an external memory (not shown), from a storage medium (such as an SD card), or from a communication network. When the video content analysis device 100 is powered on, the processor 1001 is capable of reading instructions from the RAM 1002 and executing them. These instructions form a computer program causing the processor 1001 to implement all or part of a method described in relation to [Fig. 2].

[0072] All or part of the method implemented in the video content analysis device 100, or its described variants, can be implemented in software form by the execution of a set of instructions by a programmable machine, for example a DSP (“Digital Signal Processor”) or a microcontroller, or be implemented in hardware form by a machine or a dedicated component, for example an FPGA (“Field-Programmable Gate Array”) or an ASIC (“Application-Specific Integrated Circuit”).In general, the video content analysis device 100 comprises electronic circuitry configured to implement the methods described in relation to itself as well as to remote third-party equipment, and to any other device involved in the execution of the described video content analysis method, aimed at operating one or more detections of objects and / or events in video content coming from the image capture devices 102, 104, 106 and 108. Obviously, the video content analysis device 100 further comprises all the elements usually present in a system comprising a control unit and its peripherals, such as a power supply circuit, a power supply supervision circuit, one or more clock circuits, a reset circuit, input-output ports, interrupt inputs, bus drivers. This list is not exhaustive.

[0073] Each at least one of the modules of the analysis device 100 among the enumeration next:

[0074] - module configured to determine a list of objects and / or events to be detected in one or more video contents,

[0075] - module configured to determine a list of resources necessary for detection objects and / or events to be detected,

[0076] - module configured to decompose the resources into a plurality of modules execution of a processing algorithm,

[0077] - module configured to automatically determine an optimized route of information between the different execution modules of a processing algorithm,

[0078] can be considered as an electronic circuit or a control unit operating functions of manager or supervisor of information and input data including registers of input and output data, memories, and capacities for processing this data, autonomously or combined with related resources. Each of these modules can be composed of hardware and software elements in different proportions from one module to another. The term module is not limiting and can be translated more broadly by processor, circuit, control unit, management unit, or any other synonym well known to those skilled in the art in information and signal processing or in the field of computer vision. The aforementioned modules can use common resources between them.

[0079] The invention is not limited to the embodiments described alone but relates more generally to any method for analyzing video content, executed in a video content analysis device, comprising a determination of a list of objects and / or events to be detected in one or more video contents, a determination of a list of resources necessary for the detection of the objects and / or events to be detected, from the determined list, then a decomposition of said necessary resources into a plurality of execution modules of a processing algorithm where each of said execution modules is configured to process all or part of one or more images extracted from one of the video contents, or to process information representative of a video content, obtained from one or more of the images, and at least one of the execution modules is configured to deliver information representative of a result of a detection to be carried out,and, an automatic calculation of an optimized information path between the aforementioned execution modules, aimed at minimizing the number of execution modules required to operate detections of elements among the determined list of objects and / or events to be detected. The invention also relates more generally to any device configured to implement and execute such a method.,

Claims

Claims

1. Method for analyzing video content, executed in a video content analysis device, the method comprising: - determining (S1) a list of objects and / or events to be detected in one or more video contents, - determining (S2) a list of resources necessary for the detection of said objects and / or events to be detected, - decomposing (S3) said resources into a plurality of modules for executing a processing algorithm, each of said modules being configured to process all or part of one or more images extracted from one of said video contents, or information representative of a video content, obtained from one or more of said images, and at least one of said execution modules being configured to deliver information representative of a result of a detection to be carried out,said method being characterized in that it further comprises: - automatically determining (S4) an optimized path of information between said execution modules so as to minimize the number of said execution modules required to operate detections of elements among said list of objects and / or events to be detected.,

2. A method for analyzing video content according to claim 1, wherein the steps (S2, S3) for determining (S2) a list of necessary resources and for decomposing (S3) said resources into a plurality of execution modules of a processing algorithm comprise determining or considering an index representative of a processing capacity of each of said execution modules.

3. Method for analyzing video content according to one of claims 1 and 2, in which is further included a step of determining a rate of resources used with regard to all the resources available to said video content analysis device and a step of conditioned execution of a video content analysis process depending on whether said rate exceeds or not a predetermined threshold value.

4. Method for analyzing video content according to any one of the claims- indications 1 to 3, in which said optimized data path is implemented using asynchronous messaging functions for distributed or concurrent applications, preferably according to a protocol of a type called ZeroMQ or MQTT.

5. Device for analyzing video content comprising electronic circuits including: - a module configured to determine (SI) a list of objects and / or events to be detected in one or more video contents, - a module configured to determine 5S2) a list of resources necessary for the detection of said objects and / or events to be detected, - a module configured to decompose (S3) said resources into a plurality of modules for executing a processing algorithm, each of said modules being configured to process all or part of one or more images extracted from one of said video contents, or information representative of a video content obtained from one or more of said images and at least one of said execution modules being configured to deliver information representative of a result of a detection to be carried out,said analysis device being characterized in that it further comprises: - a module configured to automatically determine (S4) an optimized path of information between said execution modules of an algorithm, so as to minimize the number of said execution modules required to operate detections of elements among said list of objects and / or events to be detected.,

6. Video surveillance system comprising one or more video content analysis modules according to the preceding claim.

7. Computer program product characterized in that it comprises program code instructions for executing the steps of the method according to any one of claims 1 to 4, when said program is executed by a processor.

8. Information storage medium comprising a computer program product according to the preceding claim.

Images