METHOD AND SYSTEM FOR AUTOMATING CAMERA MAINTENANCE OPERATIONS

Abstract

The present invention relates to a computer-implemented method for processing a data stream using an activated data processing system, said data processing system being adapted to create a plurality of dissemination channels to allow the circulation of data streams within said data processing system. Abstract Figure: 1

Description

Title of the invention: METHOD AND SYSTEM FOR AUTOMATING CAMERA MAINTENANCE OPERATIONS Field of invention

[0001] The invention relates to the field of video surveillance, also known as "video protection". More specifically, the invention relates to a method and a system for automating camera maintenance operations, said cameras belonging to a closed-circuit television system ("Closed-Circuit Television" or "CCTV" in English). State of the art

[0002] Nowadays, worldwide, the total number of cameras installed in cities and within private structures is greater than 770 million.

[0003] Such a multitude of cameras requires very significant human resources, in particular technicians, whose mission is to ensure the technical maintenance of each of these cameras.

[0004] Consequently, in order to control the costs associated with the number of technicians dedicated to the technical maintenance of a camera network, managers or end users of closed-circuit television systems are forced to use a limited number of cameras. The limitation of the number of cameras used is therefore imposed by budgetary reasons, to the detriment of the security of the geographical site concerned. Indeed, a limited number of cameras does not allow optimal video surveillance of a given geographical site to be ensured.

[0005] In the state of the art, there are solutions based on methods and systems for intelligent video analysis (in English: "Intelligent Video Analysis" or "IVA" in English).

[0006] These intelligent video analysis solutions make it possible to automate the exploitation of videos captured by cameras deployed within a closed-circuit television system, and thus reduce the human resources required for the analysis of said videos, in order to determine whether the camera is operational and functioning correctly.

[0007] Thus, thanks to the existence of these automatic analysis solutions, managers or end users of closed-circuit television systems are no longer forced to limit the number of cameras to ensure video surveillance of specific infrastructures such as a city or a private domain.

[0008] However, existing intelligent video analysis methods and systems do not do not allow for the optimization of maintenance operations for cameras used for video surveillance.

[0009] Indeed, some intelligent video analysis systems can adjust the focal length of a given camera in order to eliminate problems related to camera calibration and the lack of sharpness of the videos captured when the camera is installed on the geographical site concerned. However, this type of adjustment must be triggered manually by a technician on each new camera installed.

[0010] There are also methods for performing recurring checks on several cameras in order to analyze the videos captured by these cameras. However, these checks are based on unreliable technologies, such as image analyses based on image difference and movement in terms of pixels. These analyses are sensitive, for example, to the movements of dynamic objects such as people and vehicles, to the presence of insects on the lens, to the movements of trees, particularly due to wind, etc. In addition, some analyses are sensitive to sudden changes in illumination, such as the appearance of a reflection, or continuous changes in illumination, induced, for example, by the change in position of the sun. These types of known technologies generate a lack of robustness and therefore reliability of the video analysis system associated with these methods.Furthermore, these recurring checks are manually scheduled and allow for the assessment of a single type of technical failure. While such solutions can generate alert reports on the end user's system, producing these alert reports requires a significant amount of computation and, consequently, a considerable waste of time and significant financial loss.

[0011] Thus, such methods and systems cannot be used, in an efficient and economical manner, to ensure video surveillance of infrastructures comprising a large number of cameras to be used.

[0012] Other video analysis methods and systems exist to predict maintenance operations. This type of solution makes it possible to plan the replacement of one or more technical parts before this part produces a technical failure of the camera.

[0013] However, such solutions are limited to an intrinsic failure of the camera and cannot provide data on the overall state of use of the camera. Indeed, a malfunction of a parameter of the camera does not depend on its use and therefore cannot be predicted by current predictive maintenance techniques. These solutions may lead to replacing a camera in its entirety or interrupting the continuity of video surveillance to replace the parts concerned.

[0014] In addition, in the context of a large network of cameras, the operations of Maintenance is performed on cameras at random. Therefore, for a specific camera, possible malfunctions and technical problems are usually detected and resolved only when using this camera during video surveillance.

[0015] The state-of-the-art solutions therefore do not allow for the optimization of maintenance operations for cameras used for video surveillance, whether it is due to the high cost generated by these solutions or the unsuitable practical aspect of said maintenance operations.

[0016] Consequently, there is a need to propose new technical solutions to ensure quality video surveillance with an appropriate number of cameras, while optimizing the costs relating to the maintenance operations of these cameras. Subject of the invention

[0017] The present invention aims to meet the above-mentioned need. Thus, a first object of the present invention relates to a computer-implemented method for processing a data stream by means of an activated data processing system, said data processing system being adapted for creating a plurality of broadcast channels in order to allow the circulation of a data stream within said data processing system, each broadcast channel comprising an input connector and an output connector, said method comprising the following steps: - receipt of a request for data flow analysis by the data processing system, - determining the number of broadcast channels available within said data processing system, - if the number of available broadcast channels is greater than zero, the creation of at least one additional broadcast channel comprising an input connector and a first and a second output connector, - transmission of the data stream to be analyzed to the input connector of the additional broadcast channel, - processing of the data flow by the data processing system, - disconnecting the input connector and the first output connector from the additional broadcast channel, - transmission of the processed data stream to the second output connector of the additional broadcast channel,

[0018] so that disconnection of the input connector and the first output connector of the additional broadcast channel does not result in deactivation of the data processing system.

[0019] According to one embodiment of the invention, the step of determining the number of channels available within said data processing system comprises: - determining a total number of broadcast channels available within said data processing system for processing the data streams; - determining the number of occupied broadcast channels in the data processing system for processing data streams; and - determining the number of channels available for data processing by subtracting the number of occupied channels from the total number of available channels.

[0020] According to one embodiment of the invention, the step of creating said at least one additional broadcast channel comprises: - the creation of an additional broadcast channel with an input connector and an output connector; - connecting a TEE to the output connector of the additional broadcast channel to create the first and second output connectors.

[0021] According to one embodiment of the invention, the method further comprises: - disconnection of the input connector and the first output connector of the additional broadcast channel; - increasing the number of available channels by one unit.

[0022] According to one embodiment of the invention, the method further comprises: - determining the number of broadcast channels available within said data processing system; - if no broadcast channel is available, adding the data stream analysis request to a waiting list.

[0023] According to one embodiment of the invention, the step of adding the analysis request to a waiting list comprises: - adding a property to the data flow analysis request, associated with the priority of processing the request, relative to other data flow analysis requests.

[0024] According to one embodiment of the invention, the method further comprises: - transmitting the request for analysis of the data flow to said data processing system by means of an orchestration system, wherein said orchestration system provides instructions to said data processing system regarding the analysis of the data and the type of processing to be carried out.

[0025] According to one embodiment of the invention, the method makes it possible to determine the operating state of at least one camera, said camera being adapted to capture at least one video stream comprising a data set, said video stream relating to a determined area, and said camera also being adapted to transmit said video stream to said activated data processing system, said method comprises the following steps: - receipt of a request for video stream analysis, - capturing a specific number of video streams, - determining the number of available broadcast channels, - determining the actual number of video streams that can be transmitted to the within the available broadcast channels, said effective number of video streams corresponding to the number of available broadcast channels, - activation of the input and output connectors of each available broadcast channel, - adding a second output connector for each available broadcast channel, - transmitting the actual number of video streams to the input connector of each available broadcast channel, each input connector receiving one video stream, - processing of video streams.

[0026] A second subject of the invention relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the method described above.

[0027] A third subject of the invention relates to a computer-readable recording medium comprising instructions which, when executed by a computer, cause the latter to implement the method described above.

[0028] A fourth object of the invention relates to a system for processing data streams comprising a data processing system, said data processing system being adapted to receive the data in the form of separate broadcast channels, each broadcast channel having an input connector associated with the data input into the broadcast channel and the output connector associated with the data output from the broadcast channel, said system further comprising: - a receiver for receiving a data processing request, - a scheduler for identifying the number of broadcast channels available for processing the data, - a load management system to distribute processing requests optimally according to the filling status of the waiting lists associated with their distribution channels, - an orchestration system for creating, when at least one broadcast channel is available, an additional broadcast channel comprising a first input connector and a first output connector,

[0029] wherein the orchestration system is configured to create a second output connector for the additional broadcast channel so that the first input connector and the first output connector can be disconnected from the additional broadcast channel without causing the data processing system to be deactivated. Brief description of the drawings

[0030] The aim, object and characteristics of the invention will appear more clearly on reading the following description given with reference to the figures in which:

[0031] [Fig-1] shows a diagram of an automation system, according to a mode of rea lization of the invention;

[0032] [Fig.2] shows a diagram of the data processing system, according to a mode of realization of the invention;

[0033] [Fig. 3 - Page 1] shows the first part of a diagram relating to an automation method, according to an embodiment of the invention; and

[0034] [Fig. 3 - Page 2] shows the second part of said diagram relating to an automation method, according to an embodiment of the invention. Detailed description of the invention

[0035] The detailed description below is intended to set out the invention in a sufficiently clear and complete manner, in particular with the aid of examples, but should in no case be regarded as limiting the scope of the protection to the particular embodiments and examples presented below.

[0036] [Fig.l] shows an automation system 10 for automating camera maintenance operations according to one embodiment of the invention. Automation system

[0037] The automation system 10 comprises an acquisition system 100, a data processing system 200, a communication system 300, an alerting system 400, a planning system 500, an analysis request system 600, a load management system 800 and an orchestration system 900. Acquisition system

[0038] The acquisition system 100 makes it possible to acquire data, more particularly video streams. The acquisition system 100 comprises one or more cameras 102, 104, 106 adapted to capture video streams. For the sake of clarity, the number of cameras 102, 104, 106 shown in [Fig. 1] is limited to 3. In practice, the number of cameras 102, 104, 106 present within the automation system 100 is unlimited. The cameras 102, 104, 106 are located at determined locations within a determined geographical area delimited in advance by a user of the automation system 10. For example, the geographical area may be private property. comprising a dwelling and spaces outside said dwelling, the user wishing to monitor said private property by means of the cameras 102, 104, 106. The cameras 102, 104, 106 operate in parallel, so that each camera 102, 104, 106 can capture a video stream independently of the other cameras 102, 104, 106. The cameras 102, 104, 106 are organized to belong to the same computer network. Each camera 102, 104, 106 has an IP address (in English: "Internet Protocol") which makes it possible to identify said cameras on the computer network, said computer network using the IP protocol as a communication protocol. Any type of camera can be used, for example, the cameras can be directly IP or analog cameras whose stream is converted by an IP encoder or DVR.

[0039] Each camera 102, 104, 106 is provided with a video encoding device 108, 110, 112. As shown in [Fig.l], each of these video encoding devices 108, 110, 112 is directly integrated into the camera. Each camera 102, 104, 106 is responsible for encoding its own stream.

[0040] For the sake of completeness, it should be noted that an analog camera requires an external encoder that simultaneously provides IP transformation. In the case where the analog stream is not converted to an IP stream, the camera must be connected to a DVR, which is capable of interpreting an analog stream, rendering it over the network and then storing it.

[0041] The video encoding device 108, 110, 112 performs an encoding step which makes it possible to encode the data of the captured video stream in order to allow the transmission of said encoded video stream to a data processing system 200, detailed below, which is also present on the computer network. The video encoding device 108, 110, 112 also makes it possible to associate with the encoded video stream a plurality of initial metadata relating to the corresponding captured video stream. The initial metadata relates to the operating characteristics of the camera 102, 104, 106 having captured the video stream encoded by the video encoding device 108, 110, 112. More precisely, the initial metadata comprises three types of data. The first type of data concerns the IP address of the camera 102, 104, 106 that captured the video stream.The second type of data concerns the results of the connection test or PING test (in English: “Packet Internet Groper”) of said camera 102, 104, 106 on the computer network. The third type of data concerns the timestamp data of the video stream being encoded.

[0042] Thus, from the captured video streams, the acquisition system 100 generates a plurality of resulting video streams, each resulting video stream comprising initial metadata. Data processing system

[0043] As shown in [Fig.l], the automation system 10 also comprises a data processing system 200, connected to the acquisition system 100 via an orchestration system 900. The data processing system is adapted to process and analyze the data transmitted by the acquisition system 100. The data processing system 200 is generally known as a “streaming pipeline” in English.

[0044] The data processing system 200 is implemented on a programmable electronic machine, such as a computer, comprising a graphics card (not shown). The graphics card comprises a calculation processor (in English: "GPU" or "Graphics Processing Unit"). This calculation processor is, for example, a GPU processor of the NVIDIA® brand, based on the Hopper® architecture or later than the Hopper® architecture.

[0045] The data processing system 200 can process data in parallel using a determined quantity of distribution channels. The maximum quantity of distribution channels available is determined by the choice of the hardware constituting the data processing system 200.

[0046] As shown in [Fig.2], the data processing system 200 comprises a plurality of components detailed below. Video decoding device

[0047] Thus, the data processing system 200 comprises a decoding device 202, a multiplexing device 204, an inference determining device 206 concerning the operating state of a camera, a synchronization device 208, a demultiplexing device 210, an encoding device 212, a streaming device 214 and an on-demand recording device 216.

[0048] The decoding device 202 comprises an electronic circuit. The decoding device 202 performs a decoding step which makes it possible to convert a video stream captured by a camera 102, 104, 106 into a plurality of streams of initial images constituting said video stream. Thus, the initial data of an initial image correspond to the data of the video stream to which the initial image belongs, at a determined instant. For each initial image, the initial data define the content of said initial image. Multiplexing device

[0049] The multiplexing device 204 comprises an electronic circuit. The multiplexing device 204 makes it possible to combine in series the plurality of initial image streams. The multiplexing is done via a muxer component running directly on the GPU, thanks to its high parallelization capacity. Inference determination device

[0050] The inference determination device 206 executes in GPU memory, it takes advantage of the large number of small processors present in a GPU card making it possible to parallelize all similar operations via the use of CUDA (or “Computer Unified Device Architecture” operations). The inference determination device 206 makes it possible to predict a potential degraded operating state of one or more cameras. The inference determination device 206 makes said prediction by applying an inference method to the plurality of initial images. The application of the inference method comprises the application of a specific prediction model. The specific prediction model is a model for determining an operating state of a camera to the plurality of initial images associated with the camera 102, 104, 106 considered.The determination model is a neural network model, of the deep learning type, previously trained and validated with training images relating to correct and degraded operating states of at least one camera adapted to capture video streams of a geographical area monitored by said at least one camera.

[0051] The objective of the inference determination device 206 consists in detecting the potential existence of different types of malfunction of the camera 102, 104, 106 by applying the model for determining an operating state of a camera to a plurality of initial images associated with a determined camera 102, 104, 106.

[0052] The different types of malfunction include degraded operating states of the acquisition system 100. The degraded operating states may be generated by causes extrinsic to the acquisition system 100 or by causes intrinsic to the acquisition system 100, as detailed below.

[0053] The causes extrinsic to the acquisition system 100 may include, for example and in a non-limiting manner, the causes listed below: - First extrinsic cause: the lens of the camera 102, 104, 106 is obstructed, partially or totally, by an element external to the camera 102, 104, 106.

[0054] In this situation, if the lens of the camera 102, 104, 106 is partially obstructed, the camera 102, 104, 106 incorrectly captures the video stream of the geographical area to be monitored by said camera 102, 104, 106. Alternatively, if the lens of the camera is totally obstructed, the camera 102, 104, 106 does not capture any video stream concerning the geographical area to be monitored by said camera 102, 104, 106. It is obvious that the detection of this cause must be carried out as soon as possible to guarantee optimal operation of the automation system 10 according to the invention.

[0055] Concerning this first cause extrinsic to the camera 102, 104, 106, the inference determination device 206 carries out the prediction step on each initial image in order to obtain a prediction score. At the end of the prediction step, whatever the value of the prediction score, the inference determination device 206 generates inference metadata and combines this inference metadata with the initial metadata associated with the initial image concerned to obtain intermediate metadata. - Second extrinsic cause: the position of the camera 102, 104, 106 was modified compared to the position initially planned for said camera 102, 104, 106, by applying a physical force on the camera 102, 104, 106.

[0056] In this situation, the camera 102, 104, 106 captures a video stream which corresponds to a geographical area different from that which said camera 102, 104, 106 must monitor. It is obvious that the detection of this cause must be carried out as soon as possible in order to guarantee optimal operation of the automation system 10 according to the invention.

[0057] Concerning this second cause extrinsic to the camera 102, 104, 106, the inference determination device 206 carries out the prediction step on each initial image in order to obtain a vector representation of prediction of the content of the initial image considered.

[0058] At the end of the prediction step, the inference determination device 206 generates inference metadata and combines this inference metadata with the initial metadata associated with the initial image concerned to obtain intermediate metadata.

[0059] The causes intrinsic to the acquisition system 100 may include, for example and in a non-limiting manner, the causes listed below: - First intrinsic cause: the optical system of camera 102, 104; 106 is faulty.

[0060] In this situation, the camera 102, 104, 106 generates a resulting video stream for which the quality of the content is degraded. Such a cause may, for example, generate a resulting video stream whose content is blurred. Consequently, the quality of the initial images generated from said resulting video stream is also degraded, which results in an erroneous or even impossible prediction during the subsequent application, within the data processing system 200, of the specific determination model by the inference determination device 206 detailed below. It is obvious that the detection of this cause must be carried out as soon as possible to guarantee optimal operation of the automation system 10 according to the invention. - Second intrinsic cause: during the acquisition of the video stream, the lens of the camera 102, 104, 106 was overexposed or underexposed to a source of light and the sensor controlling the lens's light exposure is faulty.

[0061] In this situation, the camera 102, 104, 106 generates a resulting video stream for which the quality of the content is degraded. Consequently, the quality of the initial images generated from said resulting video stream is also degraded, which results in an erroneous or even impossible prediction when applying the specific determination model by the inference determination device 206. It is obvious that the detection of this cause must be carried out as soon as possible to guarantee optimal operation of the automation system 10 according to the invention. - Third intrinsic cause: following the capture of the video stream, during the step of transmitting the resulting video stream to the data processing system 200, the connection of the acquisition system 100 to the computer network was interrupted or degraded, which resulted in a loss of transmitted data, also called a video coded artifact, among the data within the resulting video stream.

[0062] In this situation, the quality of the resulting video stream transmitted to the data processing system 200 is degraded, which results in an erroneous, or even impossible, prediction when applying the specific determination model by the inference determination device 206. It is obvious that the detection of this cause must be carried out as soon as possible to guarantee optimal operation of the automation system 10 according to the invention.

[0063] Thus, concerning the causes intrinsic to the camera 102, 104, 106, the inference determination device 206 carries out the prediction step on each initial image in order to obtain a prediction score.

[0064] At the end of the prediction step, both concerning the extrinsic causes and the intrinsic causes, whatever the value of the prediction score and the vector representation of the content of the different initial images considered, the inference determination device 206 generates inference metadata and combines this inference metadata with the initial metadata associated with the initial image concerned to obtain intermediate metadata.

[0065] Within the present invention, as detailed below, specific analyses are configured by a planning system 500. As detailed below, an analysis and alerting system 400 performs specific analyses using the intermediate metadata generated by the inference determination device 206, for the purpose of detecting the extrinsic and intrinsic causes mentioned above.

[0066] The data processing system 200 comprises a synchronization device 208 which allows the application of a computer function, called a “callback” type function (English term), as known in the prior art. More precisely, the synchronization device 208 can be programmed to obtain information from one of the devices 202, 204, 206, 210, 212, 214, 216 of the data processing system 200. Thus, the synchronization device 208 can have the data currently being processed in the data processing system 200, at a specific stage of said processing of said data.

[0067] [Fig.2] shows the synchronization device 208 arranged within the system of data processing 200 such that the data received by the synchronization device 208 comes from the inference determination device 206.

[0068] The synchronization device 208 makes it possible to prepare and format the metadata obtained within the data processing system before transmitting metadata to the analysis and alert system 400 via the communication system 300, according to a first direction of circulation of the metadata. When the metadata comes from the analysis and alert system 400 via the communication system 300, according to a second direction of circulation of the metadata, the synchronization device 208 receives and formats said metadata before transmitting said metadata to the demultiplexing device 210 detailed below. The synchronization device 208 can also aggregate and prepare the requests from the alert system, in particular recording orders when said alert system detects a degraded operating state of the acquisition system 100.

[0069] The data processing system 200 also comprises a demultiplexing device 210. The demultiplexing device 210 makes it possible to separate the plurality of image streams analyzed by the analysis and alert system 400 and transmitted by said alert system 400 via the communication system 300.

[0070] The data processing system 200 also comprises a video encoding device 212. The encoding device 212 performs an encoding step which makes it possible to encode the data of the image streams analyzed and transmitted by the analysis and alert system 400 via the communication system 300. Thus, the video encoding device 212 makes it possible to group the analyzed images in order to reconstruct the different video streams such as those initially transmitted by the acquisition system 100 and each containing a plurality of images. The difference between the initial video streams and the reconstructed video streams lies in the fact that the metadata associated with the reconstructed video streams are final metadata, as detailed below, within the description of the analysis and alert system 400.The final metadata consists of the combination of the intermediate metadata and the validation and alert metadata generated by the analysis and alert system 400.

[0071] The data processing system 200 comprises a broadcasting device in streaming device 214 and a streaming device 214. These devices 214 and 216 are placed just after encoding in the execution chain of the streaming pipeline. They differ only in one point: the purpose. Indeed, as soon as the encoding is carried out, the final component of the pipeline is either a broadcasting component displaying the stream at an address in the network, where the different clients can connect to view the stream (streaming), or a component directing the video stream to a destination file, thus recording the video stream in said file. Communication system

[0072] The automation system 10 comprises a communication system 300 making it possible to interface the data processing system 200, by means of the synchronization device 208 located within said data processing system 200, and the analysis and alert system 400.

[0073] The communication system 300 makes it possible to transmit in series, that is to say in a successive manner, the initial image streams associated, respectively, with their intermediate metadata, from the synchronization device 208 to the analysis and alert system 400.

[0074] Before any transmission of a data stream to the analysis and alert system 400, the communication system 300 carries out a verification of the specific analyses in order to determine whether a complementary analysis to be carried out by the analysis and alert system 400 is programmed for the data stream concerned. If no complementary analysis is programmed, the communication system 300 transmits only the intermediate metadata to the analysis and alert system 400. If a complementary analysis is programmed, the communication system 300 transmits to the analysis and alert system 400 the intermediate metadata and the associated initial image.

[0075] After the analysis step carried out by the analysis and alert system 400 on the initial image streams and their intermediate metadata, the communication system 300 makes it possible to transmit, in series, the initial image streams analyzed and associated with the validation and alert metadata generated by the analysis and alert system 400, from the analysis and alert system 400 to the synchronization device 208. Analysis and alert system

[0076] The automation system 10 also includes an analysis and alerting system 400 which is connected to the data processing system 200 by means of the communication system 300.

[0077] The analysis and alert system 400 allows two different kinds of analyses on the data streams previously processed by the data processing system 200, as detailed below in the section relating to the planning system 500.

[0078] The analysis and alert system 400 is based on a computer programming method in Python language (registered trademark). Once the metadata and possible images are received via the communication system 300. The system 400 parallelizes the processing of the different streams received. Each stream is processed as follows: aggregation + spatio-temporal stabilization of metadata, execution of additional analyses (if programmed), generation of statistical content, generation of alert content (if alert logic is active).

[0079] Once these steps are performed in parallel for all video streams, the analysis and alert system 400 proceeds to store all metadata as well as alert and statistics content generated in the database. In the case where alert content has been generated, the analysis and alert system 400 makes a video recording request which will be directly taken into account by the recording device 216.

[0080] The aggregated, stabilized and completed metadata are then returned by the analysis and alert system 400 to the communication system 300, to then be sent to the synchronization device 208. Planning system

[0081] The automation system 10 also includes a scheduling system 500 which includes a scheduling database 502. The scheduling system 500 performs three different functions.

[0082] As shown in [Fig.l], the scheduling system is connected to the data processing system 200 via a load management system 800 and an orchestration system 900. As explained in more detail below, the load management system 800 receives instructions from the scheduling system 500 including, for example, the amount of broadcast channels needed to process the video streams. In turn, the load management system 800 sends instructions to an orchestration system 900, including, for example, the address of the camera with which the video stream was obtained, the analysis time and the type of analysis that is to be performed using the data processing system 200.

[0083] The planning database 502 does not contain any notion relating to load management, and therefore to queues. More generally, the system 500 does not evaluate the load of the system 200. This is done by the load management system 800. The system 800 therefore manages the waiting lists and transmits the information relating to a particular analysis, and this upon request from the orchestration system 900.

[0084] The planning database 502 contains the configuration data of the data processing system 200 concerning past and current configurations. In addition, the planning database 502 includes the logs of the analysis requests as well as any modifications applied to the planning. The planning database 502 also includes the start and end time of processing by the data processing system 200 as well as the allocation channel concerned.

[0085] The first function of the planning system 500 consists of programming the data processing system 200 by carrying out the following steps: - initialization of the primitives, - definition of the number of available channels, limited by the hardware computing capacity, allowing video streams or image streams to circulate simultaneously within the data processing system 200, - creation of the number of inputs and outputs of the data processing system 200 according to the number of available channels previously defined, - addition of a separation component, generally called "TEE plug-in" (English terms). Within the data processing system 200, the separation component makes it possible to create one or more parallel broadcast channels, at the output of the demultiplexing device 210, in order to divert the flow of initial images analyzed, from the demultiplexing device 210 to a parallel broadcast channel.

[0086] Thus, by placing the separation component at the output of the demultiplexing device 210, the disconnection of the video stream and the emptying of the buffer memory can be carried out in the parallel broadcast channel, and this independently of the demultiplexing device 210, that is to say without consequence on the operation of the demultiplexing device 210 which is then isolated. In a known manner, when a data stream is disconnected from a demultiplexing device and the buffer memory of said demultiplexing device is emptied, the data processing system to which the demultiplexing device belongs must be restarted before transmitting a subsequent data stream to said demultiplexing device.

[0087] However, within the present data processing system 200, the presence of the separation component at the output of the demultiplexing device 210 makes it possible to avoid a systematic restart of the data processing system 200 insofar as the subsequent data flow is transmitted to the demultiplexing device 210 and said subsequent data flow replaces by overwriting the previous data flow still present in the demultiplexing device 210. Thus, the absence of re Starting the demultiplexing device 210 makes it possible to avoid a considerable loss of time, if we consider a large number of video streams to be processed within the automation system 10. Consequently, advantageously, the data processing system 200 makes it possible to process video streams captured by the acquisition system 100 continuously, without loss of time.

[0088] The processing of the data stream is carried out in parallel in a number of broadcast channels. Typically, each broadcast channel is associated with an input connector and an output connector. As explained above, when a data stream is disconnected from a demultiplexing device and the buffer memory of said demultiplexing device is emptied, this disconnection creates an error. For this reason, the data processing in the entire data processing system to which the demultiplexing device belongs is interrupted and the system must be restarted before transmitting a subsequent data stream to said demultiplexing device.

[0089] According to the invention, each broadcast channel is associated with a first output connector and a second output connector.

[0090] This means that a distribution channel is created between an input connector and an output connector. In English, the term "PIN" is generally used to refer to said connectors. Furthermore, a second output connector is created simultaneously, for example, via a TEE. This means that the TEE is provided with a first and a second output connector.

[0091] The advantage of these measures is that when a data stream is disconnected, the input connector and the first output connector are disconnected, while the state of the second output connector remains unchanged. As a result, the occurrence of an error is avoided and data processing is not interrupted, because there is no need to restart the system after each disconnection of a stream. The advantage of these measures is that, in practice, greater flexibility is available to add and remove broadcast channels without interrupting data stream processing.

[0092] It is specified that “the primitives” are the components and variables of the data processing system 200 necessary for the proper functioning of said data processing system 200, and which, for the most part, will remain constant throughout the lifetime of the pipeline.

[0093] The initialization of these “primitives” includes, but is not limited to: the initialization of the inference components (model, model calibration files, etc.), the initialization of the callback variables 208, etc.

[0094] The second function of the planning system 500 consists of managing the data flows circulating within the data processing system 200 by carrying out the

[0095]

[0096]

[0097]

[0098]

[0099]

[0100]

[0101] next steps: - receive a request for specific analysis of a video stream by a third-party system such as a user interface associated with the automation system 10, or such as an external system belonging to the user of the automation system 10, - ensure that the video streams are processed within the data processing system 200 within optimized timeframes in order to avoid wasting time, and in an efficient manner, i.e. taking into account the maximum processing capacity of the calculation processor of the graphics card used. The third function of the planning system 500 consists of organizing the progress of the specific analyses of the video streams within the data processing system 200 in order to synchronize the deletion of a video stream at the output of the data processing system 200 and the addition of a video stream at the input of the data processing system, by carrying out the following steps: A - Request for processing a video stream: 1) if at least one channel is available, the scheduling system 500 schedules a request for processing a video stream by proceeding as follows: - adding said video stream to the input of the data processing system 200, - connection of the available channel input connector, - added an output connector on the pre-parallel broadcast channel previously created by the separation component at the output of the demultiplexing device 210. 2) if no channel is available, the scheduling system 500 schedules a request to process a video stream by proceeding as follows: - search and selection of the waiting list comprising the smallest number of pending treatment requests, - adding the request for processing said video stream to said waiting list. The scheduling system 500 thus ensures a dynamic synchronization function. B - Carrying out video stream processing within the data processing system: 1) if the waiting list contains at least one request for processing a video stream: - at the end of the processing of an existing video stream, within the system of data processing 200 and in a determined channel, the data processing system 200 operates as follows: - selection of a processing request registered in the non-empty queue of said determined channel, said selected processing request including as registration date the oldest date among the processing requests processing of said queue, using the FIFO principle (English: First-In-First-Out); - start of processing a video stream subsequent to the previous video stream whose processing has been completed, said subsequent video stream being associated with the selected processing request. Thus, the data processing system 200 starts processing the subsequent video stream as soon as the processing of the previous video stream has been completed, in a continuous manner. This continuity characteristic means that the data processing system 200 operates the processing of the different video streams without discontinuity of operation, that is to say without any waiting time and more precisely without disconnecting the input and output connections of said data processing system 200; - updating the referencing of the channel in question, in order to register the subsequent video stream as a video stream being processed in the data processing system.

[0102] This solution has a particularly advantageous technical effect, notably in that no delay is generated. Indeed, in the event of replacement of one flow by another (when the queue is not empty) no disconnection is carried out.

[0103] 2) if the waiting list does not contain a request for processing a video stream subsequent: - disconnection of the input “connector” of the channel concerned to delete the video stream whose processing has just been completed, - disconnecting the output on the additional broadcast channel previously created by the separation component and emptying the buffer.

[0104] The planning system 500 may be preprogrammed by means of a third-party system such as a user interface, said user interface being associated with the automation system 10, by means of an application programming interface (API). The planning system 500 may also be preprogrammed by means of a third-party software system such as a virtualization platform, and in particular such as a VM hypervisor (Virtual Machine) which communicates with the planning system 500 by means of an application programming interface (API).

[0105] In case a stream is disconnected, and it would not be replaced (if no analysis is queued), there are two acceptable choices, as follows:

[0106] 1) it is possible to overwrite the stream in the broadcast channel with a black image (empty), thus ensuring relatively low resource consumption on the part of the interference determination device 206. The output connector is not reconnected after the TEE. In the case of a subsequent reconnection, a new crushing is carried out, as explained above, and the output connector is reconnected.

[0107] 2) it is possible to disconnect the input connector on the multiplexer side, then clear the buffer and disconnect the output connector after the TEE. In case of a later reconnection, the input and output connectors are reconnected after the TEE.

[0108] The first choice has the advantage of simplicity of orchestration, and greater fluidity than reconnecting a flow. As for the second choice, it has the advantage of even more optimized resource use. These two types of operation can be considered, and may depend in particular on the user's preferences.

[0109] The scheduling database 502 includes the configuration data of the data processing system 200 concerning past and current configurations. The scheduling database 502 also includes the logs of the analysis requests as well as any modifications applied to the schedule. The start and end time of processing by the data processing system 200 as well as the allocation channel are included in the scheduling database 502. The scheduling system 500 has read and write access to the scheduling database system 502. Each waiting list includes data identifying the video stream to be analyzed and the content of said video stream to be analyzed. The waiting lists are not stored in a database.Only the execution and programming order are stored, the waiting lists are built dynamically according to demand (programmed or not) and computing capacity.

[0110] As mentioned above, among the various functions performed by the planning system 500, the latter manages, for each video stream, the data relating to the analyses to be carried out by the analysis and alert system 400, on these video streams.

[0111] The analyses to be carried out can be of three different types, as described below.

[0112] The first type of analyses concerns specific analyses. The specific analyses are analyses defined and recorded in the planning database 502. The specific analyses comprise the application of the inference method to the initial image streams, within the data processing system 200. The specific analyses comprise the subsequent application of an aggregation and stabilization method to the intermediate metadata, generated by the inference determination device 206, within the analysis and alert system. 400, in order to determine whether the intermediate metadata is representative of a malfunction of one or more cameras 102, 104, 106 belonging to the acquisition system 100.

[0113] The aggregation method is also applied to the vector representations for predicting the content of the different initial images considered, said vector representations for prediction being contained in the intermediate metadata. Thus, the aggregation method makes it possible to compare the vector representations for prediction with reference vector representations specific to each camera 102, 104, 106 considered in order to obtain a distance between the representations. This distance value represents the value of the offset between the image of the video stream analyzed in real time and the reference image which the system must approach to remain in operating condition.

[0114] The stabilization method makes it possible to validate the inference scores obtained over time, following the processing of the initial image streams by the inference determination device 206, within the automation system 10. Indeed, the causes of malfunction sought are, in a known manner, constant over time, insofar as these causes require human intervention on the cameras concerned. Consequently, the stabilization method makes it possible to verify whether an inference score obtained is reliable over time. All the scores and vector representations transmitted to the analysis and alert system 400 are tracked using a tracker method specific to each stream, stabilizing the results over time, which makes it possible to avoid false positives (false alerts).

[0115] The second type of analysis concerns additional analyses carried out solely within the analysis and alert system 400, optionally. The additional analyses concern a computer vision type analysis. Thus, the additional analyses concern in particular the detection of a dynamic, partial or generalized colorimetric imbalance within the initial images present within the captured video stream. Such a color imbalance can generate patches of various shapes visible on the initial images.

[0116] The third type of analysis concerns a network analysis, carried out by the analysis and alert system 400 on the results of the connection test, or PING test (in English: “Packet Internet Groper”), previously carried out within the acquisition system 100 for the camera associated with the initial images. The network analysis makes it possible to detect an anomaly concerning the computer network used by the acquisition system 100. For example, an anomaly may correspond to a resulting value of the PING test greater than a determined threshold value concerning said computer network. It should be noted that the network analysis does not require images, unlike the second type of analysis.

[0117] At the end of the specific analyses, the analysis and alert system 400 generates final metadata. This final metadata is considered valid and reliable within the automation system 10.

[0118] After applying the aggregation and stabilization methods, the analysis and alert system 400 carries out processing aimed at comparing the aggregated and stabilized results with thresholds.

[0119] The values ​​of each prediction score are compared with a previously determined threshold value of said prediction score, concerning the cause of malfunction sought. When the value of the prediction score is greater than said threshold value, this means that the operating state of the camera 102, 104, 106 associated with said initial image is degraded.

[0120] If the distance values ​​between the vector representations of the real-time flows and those of reference are greater than a threshold value, previously determined, of said comparison indicator concerning the cause of malfunction relating to the movement of a camera, this means that the camera 102, 104, 106 associated with the initial image considered has been moved.

[0121] Exceeding a threshold value results in the generation of an alert, this alert is therefore stored in the database. A video recording request is therefore issued and taken into account by the system 216.

[0122] Report generation can be governed by a schedule, or be performed in real time if configured as such. In all cases, for consistency, the reporting system directly retrieves alerts from the database and issues reports.

[0123] The analysis and alert system 400 also generates statistical data, concerning the values ​​and distance between vector representation and their evolution over time. This makes it possible to obtain an overview of the evolution of the state of the camera over time.

[0124] The API transmits the generated alert reports to the user interface associated with the automation system 10, or to the third-party software-type system having requested a direct analysis request to the direct analysis request system 600, or to the third-party electronic-type system having requested a direct analysis request to the direct analysis request system 600. Load management system

[0125] Given the information available via the planning, the processing channel assigned to an analysis request can be fixed in advance and therefore stored in the planning database 502. However, the capacity of the load management system 800 to process spontaneous analysis requests, and therefore to optimize load distribution per processing channel in real time, makes a fixed allocation of a processing channel to a scheduled analysis optional.

[0126] The load management system 800 builds a queue per processing channel and fills it following the analysis requests provided by the planning system 500 (even if the analysis requests do not have an allocation channel, the load distribution is done automatically according to the current filling level of the queues).

[0127] In addition, the load management system 800 sends the information relating to the oldest analysis request of a queue (based on the FITO (First-In-First-Out) principle) corresponding to a processing channel, according to the requests made by the orchestration system 900.

[0128] The load management system 800 sends the information relating to the analysis requests at the end of the queue to the system 400, so that the latter prepares its aggregation and stabilization processes which will be used when the system 400 receives the intermediate metadata from 300 corresponding to the current flow processed. Orchestration system

[0129] The orchestration system 900 is responsible for synchronizing the input and output connectors of the processing system 200 via disconnection / connection of the input connector and via disconnection of the additional output connector. The orchestration system 900 is therefore also responsible for stopping an analysis or replacing it with another as soon as the analysis duration has elapsed within the processing system 200. It relies on the load management system 800 to retrieve the information specific to an analysis which must be launched. Direct Analysis Request System

[0130] The automation system 10 also includes a direct analysis request system 600 that can transmit analysis requests from a third-party system (not shown) to the load management system 800 to be executed or placed in a queue. Thus, the direct analysis request system 600 operates as an interface between said third-party system, independent of the automation system 10, and the load management system 800.

[0131] The third-party system may comprise a third-party software-based system. The third-party software-based system enables a third-party scheduling system hosted by a virtualization platform such as a hypervisor or a VMS (Video Management System) to directly transmit analysis requests to the direct analysis request system 600. The third-party software-based system may communicate with the direct analysis request system 600 over a computer network, using a IP address (in English: “Internet Protocol”) specific to said third-party software system.

[0132] The third-party system may also comprise a third-party electronic system such as a home automation system, a robotic system, an alarm system, a Building Management System (BMS) or a Centralized Technical Management System (CTM). The third-party electronic system may communicate with the direct analysis request system 600 either by means of electronic connections using GPIO (General Purpose Input / Output) type ports, or on a computer network using an electronic acquisition box such as an ADAM (registered trademark) box associated with a communication protocol such as the MODBUS (registered trademark) communication protocol with a client / server type architecture. Automation process

[0133] If we consider a use of the automation system 10, this implies that the acquisition system 100, comprising one or more cameras 102, 104, 106, is installed within a geographical area subject to video protection. In addition, the automation system 10 must include analysis requests previously defined in the planning system 500 or direct analysis requests transmitted to the direct analysis request system 600.

[0134] Then, when triggering an analysis to be carried out, the automation method according to the invention comprises the following steps shown in Figure 3.

[0135] Thus, in a first step 700, the acquisition system 100 captures a video stream, by means of the camera 102. Then the video encoding device 108, associated with the camera 102, encodes the captured video stream and associates initial metadata with said video stream.

[0136] In a step 702, the acquisition system 100 transmits the video stream comprising the initial metadata to the data processing system 200.

[0137] In a step 704, within the data processing system 200, the video decoding device 202 converts the video stream into a series of images, and therefore a single image at a time T, which make up said video stream.

[0138] In a step 706, the multiplexing device 204 channels the plurality of initial images onto a single broadcast channel within the data processing system 200. This makes it possible to group reception channels into a single broadcast channel, thus enabling parallel processing of all the images at a time T of all the video streams by the inference determination device 206.

[0139] In a step 708, the inference determination device 206 applies to each initial image a previously trained prediction model, in order to generate, concerning the camera 102, on the one hand, different prediction scores respectively associated with different types of potential malfunction of a camera and on the other hand, a vector representation of each image. At the end of this step 708, the inference determination device 206 modifies the initial metadata in order to generate intermediate metadata which includes the initial metadata and inference metadata.

[0140] In a step 710, the synchronization device 208 formats the intermediate metadata.

[0141] In a step 712, the communication system 300 checks, for each initial image, whether additional analyses are programmed and operates as follows for each of the two alternatives: - if, according to a first alternative, additional analyses are programmed, then, in a step 714, the communication system 300 transmits the initial image with the corresponding intermediate metadata to the analysis and alert system 400. Then, in a step 716, the analysis and alert system 400 carries out a step of aggregating and stabilizing the intermediate metadata of all the pluralities of images constituting the initially captured video stream in order to generate validated inference scores. In a subsequent step 718, the analysis and alert system 400 carries out the additional analyses in order to modify the intermediate metadata to generate final metadata. - if, according to a second alternative, no additional analysis is programmed, then, in a step 720, the communication system 300 transmits only the intermediate metadata, relating to the initial image considered, to the analysis and alert system 400. In a subsequent step 722, the analysis and alert system 400 carries out a step of aggregation and stabilization of the intermediate metadata of the images constituting the initially captured video stream in order to generate validated inference scores.

[0142] The automation system 10 and the automation method according to the invention allow cyclical verification, by means of automated surveillance rounds, of the operating status of a camera network, for the purpose of maintaining said cameras. The automation system 10 and the automation method according to the invention make it possible to cover a plurality of camera malfunctions, such as those generally listed in the state of the art concerning video protection.

[0143] Furthermore, the use of “Deep Learning” type neural network models allows the automation system 10 and the automation method according to the invention to guarantee continuity of performance of the analyses carried out and a high level of relevance of the alerts generated and therefore of the reports generated.

[0144] Furthermore, the automation system 10 and the automation method according to the invention make it possible to optimize the human surveillance rounds carried out by a maintenance team. Indeed, thanks to the automation system 10 and the automation method according to the invention, several automated surveillance rounds can be programmed by means of the planning system 500, considering, for example, permanent priorities, seasonal priorities, while retaining the possibility of triggering a one-off analysis request concerning a specific camera.

[0145] The implementation of automated monitoring rounds and other requests for ad hoc analyses makes it possible to obtain, thanks to the automation system 10 and the automation method according to the invention, either real-time alert reports or operational reports according to a periodicity determined by the maintenance team. The alerts can result, for example, in the triggering of a visual or audible alert on the third-party system of the maintenance personnel, the arming of a calibration and correction procedure for the equipment from which the incident originates, or the change in the display of the video streams in the VMS. If the malfunction is a consequence of a continuous disturbance of the sensor parameters, a calibration and correction procedure can resolve the problem.

[0146] For the display of streams in the VMS, this means that, if an alert is generated for a particular video stream, said stream can be displayed in real time on the VMS.

[0147] In practice, the use of the automation system 10 according to the invention allows a maintenance team to intervene in a specific manner. Indeed, after receiving an operational report, the maintenance personnel can determine whether said operational report indicates the malfunction of one or more cameras. If this is the case, this means that human intervention is necessary to carry out a maintenance operation on the camera(s) whose operation is degraded.

[0148] Thus, the maintenance team can easily include the maintenance operation to be carried out in the usual intervention schedule in order to plan said maintenance operation.

[0149] Consequently, the automation system 10 and the automation method according to the invention allow a maintenance team composed of a few people to manage efficiently, and at a lower cost, a geographical area whose dimensions imply the installation of a hundred cameras, for example, to ensure video protection of said geographical area. Indeed, the automation system 10 and the automation method make it possible to avoid a significant increase in the human resources necessary to ensure the surveillance and maintenance operations of a video protection system when the dimensions of the geographical areas to be monitored increase disproportionately.

[0150] The embodiments described above are given as examples only.

Claims

Claims

1. A computer-implemented method for processing a data stream by means of an activated data processing system, said data processing system being adapted to create a plurality of broadcast channels to enable the circulation of a data stream within said data processing system, each broadcast channel comprising an input connector and an output connector, said method comprising the following steps: - receipt of a request for data flow analysis by the data processing system, - determining the number of broadcast channels available within said data processing system, - if the number of available broadcast channels is greater than zero, the creation of at least one additional broadcast channel comprising an input connector and a first and a second output connector, - transmission of the data stream to be analyzed to the input connector of the additional broadcast channel, - processing of the data flow by the data processing system, - disconnection of the input connector and the first output connector of the additional broadcast channel, - transmission of the processed data stream to the second output connector of the additional broadcast channel, so that disconnection of the input connector and the first output connector of the additional broadcast channel does not result in deactivation of the data processing system.

2. The method of claim 1, wherein the step of determining the number of channels available within said data processing system comprises: - determining a total number of broadcast channels available within said data processing system for processing data streams, - determining the number of occupied broadcast channels in the data processing system for processing data streams, and - determining the number of channels available for data processing by subtracting the number of occupied channels from the total number of available channels.

3. The method of claim 1 or 2, wherein the step of creating said at least one additional broadcast channel comprises: - creating an additional broadcast channel with an input connector and an output connector, - connecting a TEE to the output connector of the additional broadcast channel to thereby create the first and second output connectors.

4. Method according to one of the preceding claims, said method further comprising: - disconnecting the input connector and the first output connector of the additional broadcast channel, - increasing the number of available channels by one unit.

5. Method according to one of the preceding claims, said method further comprising: - determining the number of broadcast channels available within said data processing system, - if no broadcast channel is available, adding the data flow analysis request to a waiting list.

6. The method of claim 5, wherein the step of adding the analysis request to a waiting list comprises: - adding a property to the data flow analysis request, associated with the priority of processing the request, relative to other data flow analysis requests.

7. Method according to one of the preceding claims, said method further comprising: - transmitting the request for analysis of the data flow to said data processing system by means of an orchestration system, wherein said orchestration system provides instructions to said data processing system regarding the analysis of the data and the type of processing to be carried out.

8. Method according to one of the preceding claims, said method making it possible to determine the operating state of at least one camera, said camera being adapted to capture at least one video stream comprising a set of data, said video stream relating to a determined area, and said camera also being adapted to transmit said video stream to said activated data processing system, said method comprising the following steps: - receiving a video stream analysis request, - capturing a determined number of video streams, - determining the number of available broadcast channels, - determining the actual number of video streams that can be transmitted within the available broadcast channels, said actual number of video streams corresponding to the number of available broadcast channels, - activating the input and output connectors of each available broadcast channel,- adding a second output connector of each available broadcast channel, - transmitting the actual number of video streams to the input connector of each available broadcast channel, each input connector receiving a video stream, - processing the video streams.,

9. Computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the method according to one of claims 1 to 8.

10. A computer-readable recording medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to one of claims 1 to 8.

11. A system for processing data streams comprising a data processing system, said data processing system being adapted to receive the data in the form of separate broadcast channels, each broadcast channel having an input connector associated with the data input into the broadcast channel and the output connector associated with the data output from the broadcast channel, said system further comprising: - a receiver to receive a data processing request, - a planner to identify the number of broadcast channels available for data processing, - a load management system to distribute processing requests optimally according to the filling status of the waiting lists associated with their distribution channels, - an orchestration system for creating, when at least one broadcast channel is available, an additional broadcast channel comprising a first input connector and a first output connector, wherein the orchestration system is configured to create a second output connector for the additional broadcast channel so that the first input connector and the first output connector of the additional broadcast channel can be disconnected without causing the data processing system to be deactivated.

Images