A method of managing an industrial site and a system thereof

Abstract

The present invention relates to a method and system for managing an industrial site, comprising a plurality of cameras arranged relative to the industrial site capturing image data of the industrial site. The image data is transmitted to a data processing apparatus for data analysis and evaluation in a computer unit. The computer unit uses computer vision algorithms and / or feature extraction algorithms to determine a plurality of operational states, wherein the computer unit uses classifications and custom logic to determine events of an industrial process. The events are arranged in a certain order in a structured dataset.

Description

[0001] A method of managing an industrial site and a system thereof

[0002] Field of the Invention

[0003] The present invention relates to a method of managing an industrial site, capturing image data comprising a plurality of image frames of the industrial site using cameras, inputting the image to a data processing apparatus, analysing the image data in a computer unit using machine learning-based computer vision algorithms to detect objects of interests and events of an operation, storing the image data and events in a database, and determining a plurality of events.

[0004] The present invention also comprises a system configured to perform the method steps mentioned above.

[0005] The present invention further comprises a computer program configured to execute the method steps mentioned above and a computer-readable medium configured to store the computer program.

[0006] Background of the Invention

[0007] Large construction or installation projects are known to suffer from delays, design changes, unforeseen circumstances, and exceeding budgets, which can lead to claim disputes between the client and the contractor or sub-contractors. The construction or installation process is governed by a contract between the parties, which specifies terms and conditions for payments, project milestones, quality of work, breach of contract, deadline for completion of work and other information. The construction process normally consists of a plurality of various subtasks that must be completed before the construction is completed.

[0008] The contractor and sub-contractors must submit a payment claim and supporting documentation for the completed work in accordance with the contract. The client then assesses the completed work to verify the payment claim. Any discrepancies between the claimed work and the assessed work are settled in accordance with the terms laid out in the contract. Therefore, there is a desire to monitor the progress of the construction or installation, as well as the completion of subtasks and other key events. For insurance reasons, it may be desired to monitor certain areas or the entire construction site. Currently, it is known to use cameras to capture images of the monitored areas and link boxes, or data loggers, to gather operating data from the construction machinery. Workers may also use logbooks to manually log timestamps, notes, and other information related to the construction progress.

[0009] In the event of an accident, the stored image data, operating data, and logged data can then be analysed for insurance reasons. Settling disputes often requires the identification of specific tasks and the start and stop times of these tasks. This may be achieved by manually analysing the stored and logged data, which can be time consuming and labour intensive.

[0010] It is known that that objects of interest may be tracked automatically by processing the captured image data using object detection algorithms or machine learning algorithms implemented in a remote server or local data processing unit. Such algorithms can be trained and evaluated via various available programming platforms provided by software providers using either open-source datasets or custom datasets.

[0011] WO 2021 / 110226 Al discloses a method and system for monitoring an industrial site, comprising a plurality of cameras connected to a data processing apparatus. The raw image data is processed in data processing apparatus by an object detection algorithm, where the detected object is anonymised in the image data and the altered image data is stored in a database for later inspection. This solution focuses on anonymising image data so it may be stored for later analysis.

[0012] WO 2018 / 191555 Al discloses a method of recognising repeating actions of an overall assembly line process using deep learning, wherein a neutral network in communication with a long-short term memory is trained to detect actions and compare it to a benchmark process. Abnormalities in the detected actions is determined by the neural network based on the benchmark process, and video snippets of these abnormalities may be stored for later analysis. This solution is designed specifically for monitoring assembly lines for automotive parts or computer parts, and uses only two-dimensional data in the data processing.

[0013] US 11175650 B2 discloses another system for monitoring an assembly line process using sensors arranged at working stations and workers interacting with the working stations. The neural network and long-short term memory are trained to detect objects and actions from the sensors stream using deep learning, which relate to assembly steps of an overall assembly process. The workers or robots may interact with the system at the working stations, e.g., via user terminals. The neural network is configured to generate a set of performance data, which can be compared with a set of reference data. During optimization, operators may access portions of the set of data for optimising the assembly process This solution is designed specifically for real-time monitoring of the assembly of components.

[0014] US 11321944 B2 discloses a system for monitoring repeating actions in an overall process, where a neural network and long-short term memory are trained to detect actions in video data. The neural network is trained to detect the start, the end, and other details of each action based on the image frames, and to generate statistical data based on the details of the actions. This solution is designed specifically for monitoring assembly lines in a factory.

[0015] The abovementioned systems are all limited to monitoring repeating actions of an overall process, particularly of assembly lines in automotive and computer part manufacturing factories. These systems are limited for use at stationary sites, where the actions are performed at fixed stations or tables.

[0016] Other management systems for monitoring a construction site disclosed in CN 111429107 A, CA 3015879 Al, CN 111242574 A, and US 2021 / 0004591 Al.

[0017] Object of the Invention

[0018] One object of the present invention is to solve the problems of the abovementioned prior art. Another object of the present invention is to provide a method and system that allows for managing events of an industrial process at an industrial site.

[0019] Another object of the present invention is to provide a method and system that allows for detecting objects that undergo different actions during the industrial process.

[0020] Description of the Invention

[0021] One object of the present invention is achieved by a method of managing an industrial site according to claim 1, comprising the steps:

[0022] - capturing image data of at least a part of the industrial site using at least one camera, the image data comprising a plurality of image frames,

[0023] - inputting the image data to a data processing apparatus, the data processing apparatus comprising a computer unit,

[0024] - further inputting other data to the data processing apparatus, the other data being representative of conditions at or related to the industrial site,

[0025] - analysing one or more of the inputted data in the computer unit, wherein the data analysis is performed using at least one of:

[0026] - one or more computer vision algorithms configured to determine a number of operational states in the image data,

[0027] - one or more feature extraction algorithms, the feature extraction algorithms being configured to determine a number of unique features representative of a number of operational states in the other data,

[0028] - evaluating said operational states in the computer unit to determine at least one event,

[0029] - logging at least one timestamp of the at least one event,

[0030] - synchronising the at least one event with the other data inputted to the data processing apparatus to form a structured dataset, storing said structured dataset in a database.

[0031] This provides an improved method of managing an industrial process performed at an industrial site compared to conventional methods. The present method can also be used for analysing and optimizing the progress of the industrial process, or events thereof. The present method is versatile and can be adapted to different applications and to changes in the dynamic mapping at the site and is able to detect events around the entire site. Conventional methods for high-volume, in-line production sites, i.e., working stations along a production line, are not suited for use at low-volume production or assembly sites without changes to the monitoring setup as well as significant modification to the implemented computer vision algorithms. Further, conventional monitoring systems are limited to evaluate the performance at fixed workstations at factories of the automobile or electronics industry.

[0032] Here, the term “industrial process” is defined as an overall process being performed at the industrial site, wherein the industrial process is composed of a plurality of operations (or actions) performed in parallel or in a sequential order. Further, a plurality of operational states (or tasks) is performed during each operation.

[0033] Further, the term “industrial site” should in this case be understood as a site for the production, assembly, installation, or handling of large structures, or components thereof, with a size and weight unsuited for high-volume, mass production. The present system and method are particularly suited for, but not limited to, use in the energy sector, such as wind energy, the shipping (freight transport) industry, or the construction industry.

[0034] The present method uses a set of cameras to monitor the entire industrial site, or subareas of particular interest. This allows a user of the present system to visually monitor the industrial site, or sub-areas thereof. The captured image data, e.g., videos or still pictures, is inputted to a data processing apparatus for data analysis and evaluation. This allows the user to access image data of the industrial site.

[0035] Further, additional data is collected and inputted to the data processing apparatus for data storage and optional data analysis and evaluation. The additional data is representative of conditions at or related to the industrial site and may be collected from local sources and / or external sources. This allows the user to access different types of data from multiple sources.

[0036] The inputted data, or a selected subset thereof, is analysed in the computer unit using computer vision algorithms, feature extraction algorithms, or a combination thereof. The computer vision algorithms are configured to determine a number of operational states based on the captured image data. The feature extraction algorithms are configured to determine a number of unique features representative of said operational states based on the inputted additional data. This allows the computer unit to automatically detect the operational states of an operation. This also allows the computer unit to automatically detect operational states of interest.

[0037] The outputs of these algorithms are then evaluated in the computer unit using classification algorithms, custom logics, or a combination thereof to determine one or more events. The events may be operations of the industrial process or other events of interest, wherein the parameters thereof may be set up by the user or administrator of the present system. A scaling or weight value may be applied to the outputs of these algorithms, thus making the outputs better suited for clustering and / or filtering during the evaluation. This allows the computer unit to automatically detect events based on the image data and / or the inputted additional data.

[0038] The additional data and image data are synchronized in the computer unit based on the determined events to form a structured dataset. At least one timestamp of the event may be used to synchronize the different types of data. The structured dataset is stored in a dedicated database, or links may be stored together with the respective data in the database. This allows the user to access the present system and view the structured dataset of a selected event.

[0039] In one embodiment, the computer vision algorithms comprise at least one of an image classification algorithm, an optical flow algorithm, an object detection algorithm, a semantic segmentation algorithm, an action classification algorithm, a temporal action localization algorithm, or a combination thereof.

[0040] The computer vision algorithm(s) may advantageuously be implemented in the computer unit. For example, the computer vision algorithms may comprise an image classification algorithm, which may determine at least one state of process based on the content of the image frames in the image data as a whole. For example, the computer vision algorithms may comprise an optical flow algorithm, which may quantify the motion between two consecutive image frames in the image data. For example, the computer vision algorithms may comprise an action classification algorithm, which may determine at least one type of operational state based on the performed action in the stream of image frames in the image data. For example, the computer vision algorithms may comprise a temporal action localization algorithm, which may determine at least one start, end, and type of operational state based on the stream of image frames in the image data. This allows the computer unit to identify and determine the operational states using feature-based event detection.

[0041] More than one computer vision algorithm may be implemented in the computer unit, wherein the computer vision algorithms may be combined to determine the operational states. This allows for a more accurate detection of multiple operations of the industrial process.

[0042] The computer vision algorithms may also comprise an object tracking algorithm, which may track the motion of one or more objects between consecutive image frames in the image data. A one-stage or multi-stage object detection algorithm, or a semantic segmentation model, may further be implemented in the computer unit. Optionally, the computer unit may use boundary boxes or segmentation masks (regions of interest) to localise and track objects of interest within the image frames in the image data.

[0043] The computer vision algorithms may also comprise an action detection algorithm, which may determine at least one type of operational state based on the performed object actions in the image frames in the image data. The action detection algorithm may compute or combine special, temporal, and semantic features from the image data. This allows the operational states to be identified and determined in accordance with one or more object classes and action classes stored in the computer unit. This also allows various object actions to be identified and recognised within the image frames in the image data.

[0044] In one embodiment, the method further comprises the step of determining a geographical position of at least one object, and optionally tracking said at least one object throughout the industrial site.

[0045] The data processing apparatus may be configured to determine the position of one or more objects within the industrial site. The data processing apparatus may further be configured to track the motion of these objects throughout the industrial site. A coordinate system of the industrial site may be stored in the data processing apparatus for determining the (global or local) position of objects within the coordinate system. The position of an object may be determined based on the image data using a geolocalization algorithm implemented in the computer unit. For example, the geographical position of one or more cameras in combination with computer vision-based triangulation may be used to determine the geographical positions of the objects. The computer unit may then use a tracking algorithm to predict and track the positions of the objects within the image data. An example of a geo-localization algorithm is disclosed in the article “Object Tracking and Geo-Localization from Street Images” by Wilson, Daniel et al. and the article “CAPE: Camera View Position Embedding for Multi-View 3D Object Detection” by Xiong, Kaixin et al. However, other geo-localization algorithms may also be used. This allows the user to visually track objects within the industrial site using the cameras.

[0046] Optionally, a positioning system may communicate with the data processing apparatus for determining the position of one or more objects. The positioning system may transmit position data directly to the data processing apparatus, or via local positioning units on the objects. This allows objects to be tracked even if no or limited image data is available.

[0047] A local positioning unit on the object may receive beacon signals from multiple nodes of the positioning system, wherein the positioning unit determines its position based on the received beacon signals. Alternatively, multiple local positioning units of the positioning system may be arranged relative to the industrial site and the individual objects may act as mobile nodes each transmitting a beacon signal to the respective local positioning units. Optionally, a signal may be transmitted from the local positioning unit which receives the reflected signal from the respective nodes. For example, the position of the object may be determined using lateration or multi -laterati on, angulation or other known techniques. This allows objects to be tracked even if no or limited reception of global positioning signals is available.

[0048] The present method allows users of the present system to determine and track the positions of objects during various operations of the overall industrial process. This also allows for better management of the flow of materials on the industrial site, further allowing for better distribution and allocation of equipment and machinery on the industrial site.

[0049] In one embodiment, at least radio communications between at least two radio communications devices and / or audio signals picked up by at least one microphone are transmitted to the data processing apparatus, wherein the computer unit extracts voice commands and / or unique audio signals from said radio communications or audio signals using the feature extraction algorithms.

[0050] The operational states may also be determined based on the radio communications between two or more radio communications devices and / or audio signals pick up at the industrial site. The raw radio communications may be picked up by a radio scanner unit or dedicated radio unit and transmitted to the data processing apparatus and stored in the database for data analysis. One or more microphones may pick up audio signals from one or more positions at the industrial site, wherein the audio signals may include voice communications between workers, various machine sounds, audio alarms, announcements over speakers, weather related sounds and the like. The raw audio signals may be stored in the database for data analysis.

[0051] The feature extraction algorithms may be one or more speech recognition algorithms, which may detect unique voice commands in the radio communications and / or audio signals. Examples of such algorithms are disclosed in the article “Some Commonly Used Speech Feature Extraction Algorithms” by Alim, Sabur A. et al. The unique voice commands may be representative of the start, end and / or type of one or more operational states. A spectral analysis and discrete transformation of the voice signals may be used to extract the unique voice commands. Classifications for voice commands or parameters thereof may be set up by the user or administrator of the present system and stored in the computer unit. The speech recognition algorithms may be trained to detect the voice commands in one or more languages. This allows unique voice commands spoken by workers during various operations to be stored in the computer unit. This allows the computer unit to automatically determine the operational states based on the radio communications or audio signals. The feature extraction algorithms may also be one or more audio feature extraction algorithms, which may detect unique audio signals in the radio communications and / or audio signals. Examples of such algorithms are disclosed in the article “An Overview of Audio Event Detection Methods from Feature Extraction to Classification” by Ba- baee, Elham et al. The unique audio signals may be warnings, alarms, machine, or tool related sounds and / or environmental sounds. Noise reduction, filtering, and spectral analysis of the audio signals may be used to extract the unique audio signals. Classifications for unique audio signals or parameters thereof may be set up by the user or administrator of the present system and stored in the computer unit. The audio feature extraction algorithms may be trained to detect unique audio signals such as warning alarms, start or stop of engines or winches, operation of tools, or other characteristic operational sounds. This allows the computer unit to automatically detect unique audio signals associated with or affecting the performance of operations.

[0052] Optionally, a transcription (speed-to-text) algorithm may be implemented in the computer unit, wherein the transcription algorithm may perform an automatic transcription of the radio communications. The transcription may then be stored in the database for later analysis.

[0053] The radio communications may be conducted via encrypted communications links to prevent unauthorized access to the radio communications. An encrypted communications link may be established between the individual radio communications devices and / or between the radio communications devices and the data processing apparatus. An encrypted or un-encrypted link may also be established between the microphones and the data processing apparatus.

[0054] In one embodiment, telemetry data from one or more telemetry devices located within the industrial site is transmitted to the data processing apparatus, wherein the computer unit extracts at least one operational state from the telemetry data using the feature extraction algorithms.

[0055] The telemetry data of one or more telemetry devices may be transmitted to the data processing apparatus and stored in the database for data analysis. The telemetry device may receive data from one or more sensors or data logging systems located on the object. For example, the telemetry data may be representative of the operating conditions or performance of machines or tools, health conditions of workers, environmental conditions, or the like. The telemetry data may also comprise accelerations from accelerometers, movements from gyroscopes, or compass headings from magnetometers. Optionally, the position data may form part of the telemetry data. For example, the telemetry data may be standard telemetry data. Here, the term “standard telemetry data” should be understood as telemetry data that is presented in accordance with an open or globally recognised standard, such as ISO 15143-3. This allows for easy access to third- party data provided by multiple OEMs.

[0056] The feature extraction algorithm may extract one or more operational states from the telemetry data. The operational states may be representative of conditions relating to an operation of the industrial process. Reference parameters or thresholds may be set up by the user or administrator of the present system and stored in the computer unit. The selected telemetry signal may be compared to the reference parameters or thresholds in the computer unit, which may be used to detect certain signal characteristics or to determine if the telemetry signal is within safe operating conditions. The reference parameters may also be used to detect if machines or tools are turned on, or if certain actions of the machines or tool have commenced. This allows the computer unit to automatically determine unique operational states associated with an operation based on the telemetry data.

[0057] In one embodiment, telemetry data from at least one crane unit and / or image data of the at least one crane unit is transmitted to the data processing apparatus, wherein the computer unit determines at least one operational state of a crane operation using the feature extract algorithms and / or the computer vision algorithms.

[0058] The present method allows the present system to detect events that involves the use of a crane unit, since crane operations are sensitive to environmental and load conditions and need to be monitored carefully to prevent any accidents.

[0059] Telemetry data of the crane unit may thus be transmitted to the data processing apparatus, wherein the telemetry data is analysed in the computer unit. The feature extraction algorithms implemented in the computer unit may be used to determine the operational states of a crane operation.

[0060] If the crane unit is within the field of view of one or more cameras, then the computer unit may also analyse the image data using the computer vision algorithms. The computer vision algorithms may thus be used to also determine the operational states of the crane operation.

[0061] In one embodiment, meteorological data from a meteorological unit or system is transmitted to the data processing apparatus, optionally the computer unit extracts at least one environmental feature from the meteorological data using the feature extraction algorithms.

[0062] The meteorological data from one or more meteorological units or systems may be transmitted to the data processing apparatus and stored in the database. The meteorological data may be representative of the environmental conditions relating to the industrial site. For example, the meteorological data may comprise wind speed, temperature, lightning, humidity, precipitation, visibility, cloud cover, air pressure, wave height, wavelength, wave dwell, wave direction, or other relevant weather, wind data, or wave data.

[0063] Optionally, the computer unit may extract one or more environmental features of the meteorological data using the feature extraction algorithms. The environmental feature may be related to an operational state of an operation. Reference parameters or thresholds may be set up by the user or administrator of the present system and stored in the computer unit. The selected meteorological signal may be compared to the reference parameters or thresholds in the computer unit, which may determine if the meteorological signals are within safe operating conditions. This allows the computer unit to automatically determine meteorological features influencing an operational state based on the meteorological data. This further allows users to evaluate the root-cause of abnormal situations, such as delays, failures, or accidents, in an easier and less data-heavy investigation. The meteorological data may be actual historical weather data or weather forecasts hereof, as planning decisions are often made based on the forecast. Actual historical weather data can be used to determine e.g., abort scenarios, in which actual weather conditions exceeded safe working conditions despite the weather forecast predicting safe weather conditions.

[0064] In one embodiment, the method further comprises logging additional data into the data processing apparatus using a user terminal, the logged data is representative of conditions related to a selected operational state.

[0065] The present method allows workers to access the present system and manually inputting additional data relating to one or more operational states of the industrial process. The additional data may be logged into the present system via a dedicated user interface. For example, the additional data comprise image data (videos or still pictures), position data, text messages, measurements, or the like. The user interface may be set up so that the worker can select text messages from a list or menu, upload image data or position data, record measurements or voice messages, or manually enter text.

[0066] The computer unit may automatically link the additional data to a dedicated operational state or operation based on the logged data. Alternatively, the worker may select a dedicated operational state or operation when logging data.

[0067] The dedicated user interface may be implemented on a mobile or handheld device, a user terminal, or a computer device in communication with the data processing apparatus. Alternatively, the worker may access a web-based user interface on the data processing apparatus.

[0068] In one embodiment, the method further comprises at least one of:

[0069] - tracking the movement of a first object relative to at least a second object or a fixpoint,

[0070] - determining a first event and at least a second event, wherein both the first and second events involve the same object(s),

[0071] - determining the positions of a machine or tool and a selected object relative to a dedicated point of operation. The path of objects moving between various locations at the industrial state may be tracked by the computer unit using the image data and / or the position data. Further, multiple objects of the same type or different types may be tracked relative to each other or relative to one or more fixpoints. Computer vision algorithms may be used to identify types of objects. Optionally, the distance between objects and / or between a selected object and a fixpoints may be calculated based on the image data and / or the position data. This allows the user of the present system to identify the current position of objects within the industrial site, and to determine if objects are spaced too far apart or too close relative to each other. This also allows the user of the present system to evaluate the path of movement of objects and optimize the flow within the industrial site.

[0072] Further, event classifications may be used to identify types of events involving a particular object, such as dedicated operations and operational states thereof. Optionally, object state classifications may be used to identify types of object states. The event classifications may also be used to identify types of events that involves larger, assembled object units. This allows the user of the present system to monitor various actions performed on the objects during one or more operations of the industrial process. The various actions may be loading or unloading of objects, preparation of objects before next operation, lifting or lowering of objects into alignment or assembly of objects into a larger object unit.

[0073] The computer unit may also determine the position of a machine or tool relative to a dedicated point of operation and / or to a selected object based on the image data and / or the position data. This allows the user of the present system to evaluate the placement of various of machines or tools and objects relative to the dedicated point of operation. This reduces waiting time and allows for a smoother operation.

[0074] In one embodiment, the image data is anonymised using an anonymisation algorithm implemented in the data processing apparatus, wherein the anonymised image data is stored in a database.

[0075] The raw image data may be stored locally at the industrial site, e.g., in the local node or in the camera. The raw image data may be encrypted before being transmitted to the data processing apparatus. The encrypted image data may then be decrypted, and the raw image data may be anonymised using an anonymisation algorithm implemented in the computer unit. The anonymised image data may then be stored in another database, e.g., located at the industrial site or at a remote location. This allows the present method to comply with the data protection law.

[0076] The data processing and data storage may thus be performed off-site, thereby reducing the amount of local data processing and reducing the installation costs of the present system. Alternatively, the data processing and data storage may be performed on-site, thus reducing the amount of data transmission. Users may then access the local or remote data processing apparatus via the user terminal using dedicated user profiles.

[0077] An initial data processing, e.g., the anonymisation of the image data, may be performed in the respective camera units before transmitting the image data to the data processing apparatus. Optionally, the computer vision algorithms and / or the feature extraction algorithms may further be implemented in a local data processing apparatus or in the camera units. This reduces the need for remote data processing as any less demanding data processing algorithms can be performed locally.

[0078] In one embodiment, a first timestamp and a second timestamp of the at least one event are logged, wherein a time period of the at least one event is calculated based on the first and second timestamps.

[0079] The computer unit may automatically log one or more timestamps for each event using the meta data of the image data. For example, the computer unit may determine a start timestamp, an end timestamp, and any other timestamps of the event. The time period for the event may be calculated as the difference between the start timestamp and the end timestamp. The respective timestamps may be stored in the database together with the structured dataset.

[0080] The timestamps may also be used to identify the image segment of that event in the stored image data. Alternatively, a copy of the image segment may be stored together with the structured dataset. Alternatively, the computer unit may use the internal clock of the data processing apparatus to log the timestamps of the events. The internal clock may also be used to verify or synchronize the timestamps of one or more of signals inputted to the data processing apparatus. Optionally, the timestamps of one inputted signal may be used to verify the timestamps of another inputted signal. This allows for the correction of incorrect or missing timestamps in the inputted signals.

[0081] In one embodiment, the method further comprises steps of:

[0082] - updating a training dataset based on at least the events, the image data, or the inputted data, and

[0083] - retraining the algorithms implemented in the computer unit using the updated training dataset.

[0084] A training dataset may be used to train the algorithms implemented in the computer unit using machine learning to automatically detect and identify various events and objects. The training dataset may be historic data from a previous industrial site or previous data from the industrial site. The training dataset may comprise the events, the image data, the inputted data, or any combination thereof. An administrator of the present system may evaluate the output of the computer unit after each training run and correct any errors and / or missed objects or events in the output. Once the evaluation is completed, a subsequent training run may be performed. The training section may be repeated until an acceptable level of performance is achieved.

[0085] The training dataset may be updated upon request, at regular intervals or after each training run. New objects and / or events may be added to the classifications, and / or the parameters for existing objects or events may be adjusted. The algorithms implemented in the computer unit may thus be trained to detect objects and / or events unique for a particular industrial site, as the size and complexity of each industrial site may vary during the overall industrial process. The classifications may comprise standardized classes, customized classes, or a combination thereof, which may be updated during evaluation and training of the computer unit.

[0086] In one embodiment, the events of the industrial process are arranged in a chronological or predetermined order in the structured dataset. The computer unit may arrange the events in accordance with a predetermined order descriptive of the industrial process. Alternatively, the events may be arranged in a chronological order. This allows users to identify if events are performed in an incorrect order, or if events are not started or waiting for the completion of other events. This also allows user to easy identify any missing events of a particular sequence of operations.

[0087] The data received by the data processing apparatus may be processed in batches or in real-time, and the structured dataset may be updated continuously based on the detected events. This allows users of the present system to view the ongoing events and the completed events, thus allowing for an easy status of the progress of the industrial process.

[0088] In one embodiment, a statistical analysis is performed by the computer unit on the structured dataset using the first and second timestamps and / or the time period, wherein the results of the statistical analysis are stored in the database.

[0089] Optionally, a statistical analysis may be performed by the computer unit on the stored structured dataset using the timestamps and / or the time period of the events, where the results of the statistical analysis may also be stored in the database for later analysis. The results may include averaging times, time periods between selected events, and other relevant statistical data for each operation and the overall industrial process. This allows users of the system to monitor the time spend for each event as well as when a particular event was started, stopped, or ended. One or more parameters of the statistical data may be used as references for detecting abnormal events, and such abnormal events may be used for later analysis and optionally retraining of the data processing apparatus.

[0090] The structured dataset and / or the results of the statistical analysis may be presented to users of the present system as a report or in a dedicated graphic user interface.

[0091] One object of the present invention is achieved by a system, as mentioned in claim 14, comprising the steps:

[0092] - at least one camera configured to capture image data of at least a part of the industrial site, the image data comprising a plurality of image frames, - a data processing apparatus comprising a computer unit, wherein the cameras are connected to the data processing apparatus via a communications link for inputting the image data to the data processing apparatus,

[0093] - the data processing apparatus being configured to further receive other data from other sources via another communications link, the other data being representative of conditions at or related to the industrial site,

[0094] - wherein the computer unit of the data processing apparatus is adapted to execute the steps of the method described above.

[0095] This provides an improved system for managing an industrial process performed at an industrial site compared to conventional methods. The present system can also be used for analysing and optimizing the progress of the industrial process, or events thereof. The configuration of the present system is versatile and can be adapted to different applications and to dynamic changes in the mapping at the site. Conventional monitoring systems for stationary production sites are not suited for dynamic sites without changing the system setup and modifying the algorithms in the computer unit.

[0096] The cameras are arranged relative to the industrial site to capture image data of the industrial site, or sub-areas thereof, wherein the resolution and frame rate are selected based on the application and type of industrial site. Each camera is connected to the data processing apparatus via a first communications link, e.g., an encrypted communications link. The data processing apparatus comprises at least of a computer unit and a database in communication with computer unit, wherein a computer program is implemented on the computer unit and configured to execute the method step described above.

[0097] One or more computer vision algorithms are implemented in the computer unit for determining a plurality of operational states of an operation based on the image data. The data processing apparatus uses classification algorithms and custom logics to determine an operation of the industrial process based on the operational states.

[0098] In one embodiment, the data processing apparatus is configured to communicate with a positioning system via a communications link, wherein the positioning system is configured to input position data to the data processing apparatus. The data processing apparatus may be configured to determine the position of one or more objects within a coordinate system defining the industrial site. The data processing apparatus may further be configured to track the motion of these objects throughout the industrial site using the position data and / or the image data. The data processing apparatus may communicate with a server of a third-party provider, or an integrated positioning system of the present system.

[0099] The positioning system may be a Global Navigation Satellite System (such as GPS or GLONASS), a Local Positioning System (such as a radio, BLE, WIFI, UWB, or LIFI- based positioning system) or an Indoor Positioning System (such as HPPS).

[0100] In one embodiment, the computer unit is configured to combine the image data from multiple cameras into a multi-dimensional mapping of the industrial site.

[0101] A plurality of cameras may be positioned relative to the industrial site and angled so that they cover the monitored area from different angles. The inputted image data may be combined in the computer unit to generate a multi-dimensional mapping of the monitored area, e.g., a 2D or 3D map. This allows for a better tracking of the objects as the installation often requires the objects to be moved in multiple directions within the coordinate system before being mounted or assembled to another object or structure.

[0102] In one embodiment, the system further comprises:

[0103] - at least two radio communications devices configured to communicate with each other via a radio communications link, the radio communication between the radio communications devices being transmitted to the data processing apparatus, and / or

[0104] - at least one microphone connected to the data processing apparatus, wherein the at least one microphone is configured to pick up audio signals from at least one position at the industrial site.

[0105] The data processing apparatus may comprise a radio scanner unit configured to scan the radio frequencies used by a plurality of radio communications devices to conduct radio communications between workers. The radio scanner unit may be configured to tune to a particular radio channel being used and pick up the raw radio communication between the radio communications devices. Alternatively, a dedicated radio unit may be set up to access the radio channel being used. The radio communication between the radio communications devices may then be stored in the database of the data processing apparatus. An encryption key may be entered into or stored in the radio scanner unit to access the radio frequencies if encrypted radio channels are used.

[0106] Alternatively, or additionally, one or more microphones may pick up audio signals from one or more positions at the industrial site, wherein the audio signals may include voice communications between workers, various machine sounds, audio alarms, announcements over speakers, weather sounds, and the like. The audio signals picked up by the microphones may be stored in the database of the data processing apparatus.

[0107] A speech recognition algorithm may be implemented in the computer unit to extract unique voice commands from the radio communications and / or the audio signals. The computer unit may use these unique voice commands to further determine the operational states of an operation. Further, an audio feature extraction algorithm may be implemented in the computer unit to extract unique audio signals associated with the performance of an operation and / or other characteristic audio signals that indirectly or directly affect the performance of an operation. For example, classes comprising machine operating sounds may be used for determining machine operating conditions. For example, classes comprising weather related sounds may be used for determining environmental operating conditions.

[0108] Optionally, a transcription (speech-to-text) algorithm may be implemented in the computer unit to generate a transcription of the radio communication. The transcription may be stored in the database for later analysis.

[0109] In one embodiment, the system further comprises a plurality of telemetry devices located within the industrial site and / or at least one meteorological unit or system, wherein the telemetry devices and / or the meteorological unit or system are configured to communicate with the data processing apparatus via one or more communications links.

[0110] The data processing apparatus may be configured to communicate with a plurality of telemetry devices located within the industrial site for receiving telemetry data. Alternatively, the telemetry data may be provided from a third-party provider. A feature extraction algorithm may be implemented in the computer unit to extract operational states from the telemetry data, such as crane based operational states. The data processing apparatus may establish a wired or wireless communications link with the telemetry devices or a server of the third-party provider. These operational states may also be used to determine one or more operations of the industrial process.

[0111] Further, the data processing apparatus may be configured to communicate with a meteorological unit or system located relative to the industrial site for receiving meteorological data. A third-party provider may provide the meteorological data. The data processing apparatus may establish a wired or wireless communications link with the meteorological unit or system or a server of the third-party provider. The meteorological data may be stored in the database and combined with the image data and / or position data to form a structured dataset.

[0112] In one embodiment, the industrial site is an installation or shipping vessel, a shipping terminal, or an installation or construction site, a heavy loading or unloading area, or a production or pre-assembly factory or area.

[0113] The present method is suited for monitoring industrial sites where multiple objects are moved around and different events involving these objects are performed, particularly sites where repetitive events are performed but the location for these events may change during the overall process.

[0114] The industrial site may be a dynamic site, such as an installation or shipping vessel, where objects must be loaded and stored in predetermined pattern for optimal usage of the available storage space, and where the objects must be installed or unloaded in a particular order. The dynamic site may also be a shipping terminal (such as airports, seaports, or standalone inland terminals), where objects must be loaded or unloaded and optionally stored in predetermined pattern, and where the movement of objects must be coordinated with the loading / unloading process to reduce idle time. The dynamic site may also be a construction or installation site where objects are temporary stored between usage or installed directly upon arrival, and where the mapping of the site changes during the construction process. The dynamic site may also be a site (such as a temporary site) where various tasks or operations are performed within dedicated areas under controlled environments, and objects are moved between said areas during the industrial process. For example, a heavy loading or unloading area for offshore or onshore transport, or a production or pre-assembly factory or area for large, heavy components. For example, the dynamic site may be a base port for the handling and loading of offsite components before installation. Optionally, the base port may comprise local assembly and / or production facilities so that the offshore components can be pre-assembled and / or produced onsite at the base port.

[0115] The present system can thus be used at sites where objects are moved around and where there is need for monitoring repeatable events, allows the user of the present system to easily track and monitor events and objects.

[0116] One object of the present invention is achieved by a computer program, as mentioned in claim 20, comprising instructions which, when loaded and run on the system described above, causes the data processing apparatus to execute the method steps described earlier.

[0117] The present method is implemented into the data processing apparatus, wherein the computer unit is configured to execute the steps of the present method. This allows the use and training of artificial intelligence, such as neutral networks (e.g., FNN, CNN, RNN, attention networks, DNN or SNN) using machine learning to automatically determine the events of the industrial process.

[0118] One object of the present invention is achieved by a computer-readable medium, as mentioned in claim 21, having stored thereon the computer program described above.

[0119] The present computer program is implemented on the computer-readable medium of a remote server (or server network) or a local computer unit, thus allowing the data processing and storage to be adapted to desired site configuration and application. Alternatively, a hybrid version of the data processing apparatus may be used where parts of the computer program may be implemented in the remote server and parts of the computer program may be implemented on the local computer unit. In the hybrid version, the data analysis, the data storage, the data evaluation, and the data access and interaction may be implemented on and executed by one or more remote servers or server networks. Thus, allowing the present data processing apparatus to be configured as a local-based data service, a remote- or cloud-based data service, or a hybrid data service thereof.

[0120] Description of the Drawing

[0121] The present invention is described by example only and with reference to the drawings, wherein:

[0122] Fig. 1 shows a block diagram of a first embodiment of the method according to the present invention,

[0123] Fig. 2 shows a first embodiment of the present system according to the present invention,

[0124] Fig. 3 shows a second embodiment of the present system,

[0125] Fig. 4 shows a third embodiment of the present system,

[0126] Fig. 5 shows a fourth embodiment of the present system,

[0127] Fig. 6 shows an example of synchronizing the image data and the other data,

[0128] Fig. 7 shows an example of the structured dataset stored in the database,

[0129] Fig. 8 shows an exemplary application of the present system for an offshore site, and

[0130] Fig. 9 shows an exemplary application of the present system for an onshore site.

[0131] In the following text, the figures will be described one by one, and the distinct parts and positions seen in the figures will be numbered with the same numbers throughout the different figures. Not all parts and positions indicated in a specific figure will necessarily be discussed together with that figure.

[0132] Detailed Description of the Invention

[0133] Fig. 1 shows a block diagram of a first embodiment of the method according to the present invention. The method is adapted to manage an industrial site where the mapping of the industrial site may change during the industrial process.

[0134] A number of cameras are positioned relative to the industrial site, each of which captures image data 1 of the industrial site. The frame resolution, shutter speed and other settings of the cameras are optionally adapted to the industrial process performed at the industrial site. The image data is inputted to a data processing apparatus via a communications link, e.g., an encrypted communications link, wherein the image data is processed and analysed 2 as mentioned later.

[0135] Optionally, other types of data 3 are further inputted to the data processing apparatus together with the image data 1. The other data 3 and the image data 1 are then processed and analysed 2 in the computer unit of the data processing apparatus. The other data 3 are representative of conditions at or related to the industrial site, and some of the other data 3 may be provided by third-party providers.

[0136] Here, the other data 3 includes, but is not limited to, location data 3a, telemetry data 3b, acceleration and gyroscopic data 3c, meteorological data 3d, radio communications 3e and logged data 3f. Other types of other data may be inputted to the data processing apparatus depending on the application of the present system.

[0137] The location data 3a is inputted via a positioning system, such as a global or local positioning system, in communication with the data processing apparatus. The location data 3a is indicative of the locations of one or more objects within the industrial site.

[0138] The telemetry data 3b and / or the acceleration and gyroscopic data 3c may be inputted directly to the data processing apparatus from sensors or other devices arranged on objects located within the industrial site. The telemetry data 3b and / or the acceleration and gyroscopic data 3c may also be inputted from a third-party provider, which manages the operation of one or more objects, such as cranes, vessels, machines, or the like, within the industrial site.

[0139] Optionally, the location data 3a, the telemetry data 3b, and / or the acceleration and gyroscopic data 3c may be supplied by the same third-party provider.

[0140] The meteorological data 3d is inputted via a local meteorological unit or a third-party provider to the data processing apparatus. The meteorological data 3d is representative of the environmental conditions relating to the industrial site. The meteorological data 3d includes, but is not limited to, historical weather data, and / or weather forecasts. The radio communications 3e between radio communications devices located within the industrial site is further inputted, e.g., directly, to the data processing apparatus. The radio communications 3e is optionally encrypted to prevent unauthorised access to the radio communication.

[0141] Workers at the industrial site can manually log further data relating to the conditions at the industrial site or to operations performed at the industrial site.

[0142] The inputted image data 1 and the inputted other data 3 are analysed 2 by a computer unit in the data processing apparatus to determine a number of operational states within the image data and the other types of data 3. The operational states are preferably determined by the computer unit using computer vision algorithms, feature extraction algorithms, or a combination thereof. Optionally, the inputted data 1, 3 are filtered and pre-processed into signals or signal ranges suitable for the subsequent data analysis.

[0143] The operational states are then evaluated 4 by the computer unit to determine a number of events 4a using classification algorithms, custom logics, or a combination thereof. The events 4a determined by the computer unit are then synchronised with the image data and the other data to form a structured dataset 4b. The structured dataset 4b along with the events 4a determined by the computer unit are then stored in a database 5.

[0144] Users of the present system are then able to access the data processing apparatus and view the results 6 of the evaluation process. The structured dataset 4b together with the events 5 may be presented to the user as a report 6a or in a dedicated graphic user interface 6b. The user or administrator of the present system is further able to input data from third-party providers via a dedicated API interface 6c.

[0145] Fig. 2 shows a first embodiment of the system according to the present invention. Here, the data processing apparatus is configured as a hybrid data service, where parts of the data processing 2’ (ref. data analysis) are performed locally while other parts of the data processing 2” are performed remotely.

[0146] The image data 1 and other data 3 are captured or measured locally 7 using cameras, sensors, or other data recording devices 8a, 8b. The image data 1 and other data 3 are inputted to a local data processing unit where the inputted data 1, 3 are at least analysed 2’ by a local computer unit in the local data processing unit. The data outputted from the local data processing unit is transmitted to a remote data processing unit, wherein the outputted data is at least evaluated remotely 9 for determining the events 4a.

[0147] This remote data processing unit may further receive data 8c inputted from third-party providers. This data 8c is then analysed by the remote computer unit in the remote data processing unit to determine the operational states.

[0148] The data outputted from the local data processing unit together with the data 8c from third-party providers are then evaluated by the remote computer unit to determine the events 4a and generate the structured dataset 4b. The events 4a and structured dataset 4b are then stored in a remote database 5.

[0149] Users are then able to access the remote data processing apparatus by a dedicated API interface using their respective user profiles. The user can then view or download the events 4a and structured dataset 4b by transmitting data requests 10 to the remote data processing apparatus. The user may request corrections or changes to the structured dataset 4b via the API interface.

[0150] Fig. 3 shows a second embodiment of the present system. Here, the data processing apparatus is configured as a remote data service, where the data processing 2 is performed remotely.

[0151] The image data 1 and other data 3 are captured or measured locally 7’ using cameras, sensors, or other data recording devices 8a, 8b. The image data 1 and other data 3 are transmitted to the remote data processing unit, wherein the received data together with the data 8c inputted from third-party providers are analysed and evaluated remotely 9”.

[0152] The computer unit in the remote data processing unit determines the events 4a and generates the structured dataset 4b. The events 4a and structured dataset 4b are then stored in the remote database 5. Fig. 4 shows a third embodiment of the system. Here, the data processing apparatus is configured as an alternative remote data service, where the data storage 5 is performed in a remote database 5’ separate from the data processing 2.

[0153] The computer unit in the remote data processing unit determines the events 4a and generates the structured dataset 4b. The events 4a and structured dataset 4b are then transmitted to the remote database 5’ for data storage. The remote database 5’ is in communication with the remote data processing apparatus for accessing the stored data.

[0154] Fig. 5 shows a fourth embodiment of the system. Here, the data processing apparatus is configured as a local data service, where the data processing 2 is performed locally.

[0155] The image data 1 and other data 3 are captured or measured locally 7” using cameras, sensors, or other data recording devices 8a, 8b. The image data 1 and other data 3 are inputted to the local data processing unit, wherein the inputted data together with the data 8c inputted from third-party providers are analysed and evaluated locally 7”.

[0156] The computer unit in the local data processing unit determines the events 4a and generates the structured dataset 4b. The events 4a and structured dataset 4b are then stored in a local database 5”.

[0157] Users are then able to access the local data processing apparatus remotely 9”’ via a dedicated API interface using their respective user profiles. The user can then view or download the events 4 and structured dataset 4b by transmitting a data request 10 to the remote data processing apparatus. The user may request corrections or changes to the structured dataset 4b via the API interface.

[0158] Although different versions of the system are illustrated in Figs. 2-5, other configurations of the system may be used.

[0159] Fig. 6 shows an example of synchronizing the image data 1 and the other data 3. Here, the image data 1 and the other data 3 are synchronised using a timestamp signal 11. The data associated with each event 4a in the structured dataset 4b are synchronised using this timestamp signal 11. One or more timestamps 1 la are logged and linked to each event 4a. Thus, allowing the user to view at least the start, the end, and the time period for each event 4a.

[0160] Fig. 7 shows an example of the structured dataset 4b stored in the database 5. Here, the dataset 4b is presented as a matrix where the relevant inputted data 1, 3 for each event 4a are listed together with the number of events 4a determined by the computer unit. The matrix may be changed or altered upon request from the users, wherein the administrator of the present system may control the configuration of the matrix.

[0161] Fig. 8 shows an exemplary application of the present system for an offshore site. Here, the industrial site is an offshore site, such as an installation vessel 12 for erection of a wind turbine 13. The installation vessel 12 comprises a crane unit for lifting the various wind turbine components into position relative to each other.

[0162] Cameras 14 are positioned at predetermined locations on the offshore site, wherein the cameras 14 are used to monitor areas of interest. Other sensors (not shown) are further arranged at the offshore site for inputting other data 3 to the data processing apparatus 15.

[0163] Here, the operation of at least the crane unit can be monitored using the cameras 14 and the sensors (not shown), wherein image data and measured data of the crane operations are transmitted to the data processing apparatus 15 for data analysis and evaluation. Thereby allowing at least events involving the use of the crane unit to be determined by the computer unit in the data processing apparatus 15.

[0164] Fig. 9 shows an exemplary application of the present system for an onshore site. Here, the industrial site is an onshore site, such as an erection site of a wind turbine 13. The installation involves the use of a crane unit 16 for lifting the various wind turbine components into position relative to each other.

[0165] Cameras 14 are positioned at predetermined locations at onshore site, wherein the cameras 14 are used to monitor areas of interest. Other sensors (not shown) are further arranged at the onshore site for inputting other data 3 to the data processing apparatus 15. Here, the operation of at least the crane unit 16 can be monitored using the cameras 14 and the sensors (not shown), wherein image data and measured data of the crane operations are transmitted to the data processing apparatus 15 for data analysis and evaluation. Thereby allowing at least events involving the use of the crane unit 16 to be deter- mined by the computer unit in the data processing apparatus 15.

Claims

CLAIMS1. A method of managing an industrial site, comprising the steps:- capturing image data of at least a part of the industrial site using at least one camera, the image data comprising a plurality of image frames,- inputting the image data to a data processing apparatus, the data processing apparatus comprising a computer unit,- further inputting other data to the data processing apparatus, the other data being representative of conditions at or related to the industrial site,- analysing one or more of the inputted data in the computer unit, wherein the data analysis is performed using at least one of:- one or more computer vision algorithms configured to determine a number of operational states in the image data,- one or more feature extraction algorithms, the feature extraction algorithms being configured to determine a number of unique features representative of a number of operational states in the other data,- evaluating said operational states in the computer unit to determine at least one event,- logging at least one timestamp of the at least one event,- synchronising the at least one event with the other data inputted to the data processing apparatus to form a structured dataset, storing said structured dataset in a database.

2. The method according to claim 1, characterised in that the computer vision algorithms comprises at least one of an image classification algorithm, an optical flow algorithm, an object detection algorithm, an action classification algorithm, a temporal action localization algorithm, or a combination thereof.

3. The method according to claim 1 or 2, characterised in that the method further comprises the step of determining a geographical position of at least one object, and optionally tracking said at least one object throughout the industrial site.

4. The method according to any one of claims 1 to 3, characterised in that at least radio communications between at least two radio communications devices and / or audio signals picked up by at least one microphone are transmitted to the data processing apparatus, wherein the computer unit extracts voice commands and / or unique audio signalsfrom said radio communications or audio signals using the feature extraction algorithms.

5. The method according to any one of claims 1 to 4, characterised in that telemetry data from one or more telemetry devices located within the industrial site is transmitted to the data processing apparatus, wherein the computer unit extracts at least one operational state from the telemetry data using the feature extraction algorithms.

6. The method according to claim 5, characterised in that telemetry data from at least one crane unit and / or image data of the at least one crane unit is transmitted to the data processing apparatus, wherein the computer unit determines at least one operational state of a crane operation using the feature extract algorithms and / or the computer vision algorithms.

7. The method according to any one of claims 1 to 6, characterised in that meteorological data from a meteorological unit or system is transmitted to the data processing apparatus, optionally the computer unit extracts at least one environmental feature from the meteorological data using the feature extraction algorithms.

8. The method according to any one of claims 1 to 7, characterised in that the method further comprises logging additional data into the data processing apparatus using a user terminal, the logged data is representative of conditions related to a selected operational state.

9. The method according to any one of claims 1 to 8, characterised in that the method further comprises at least one of:- tracking the movement of a first object relative to at least a second object or a fixpoint,- determining a first event and at least a second event, wherein both the first and second events involve the same object(s),- determining the positions of a machine or tool and a selected object relative to a dedicated point of operation.

10. The method according to any one of claims 1 to 9, characterised in that the image data is anonymised using an anonymisation algorithm implemented in the data processing apparatus, wherein the anonymised image data is stored in a database.

11. The method according to any one of claims 1 to 10, characterised in that a first timestamp and a second timestamp of the at least one event is logged, wherein a time period of the at least one event is calculated based on the first and second timestamps.

12. The method according to any one of claims 1 to 11, characterised in that the method further comprises steps of:- updating a training dataset based on at least the events, the image data, or the other data, and- retraining the algorithms implemented in the computer unit using the updated training dataset.

13. The method according to any one of claims 1 to 12, characterised in that the events of the industrial process are arranged in a chronological or predetermined order in the structured dataset.

14. A system for managing an industrial site, comprising the steps:- at least one camera configured to capture image data of at least a part of the industrial site, the image data comprising a plurality of image frames,- a data processing apparatus comprising a computer unit, wherein the cameras are connected to the data processing apparatus via a communications link for inputting the image data to the data processing apparatus,- the data processing apparatus being configured to further receive other data from other sources via another communications link, the other data being representative of conditions at or related to the industrial site,- wherein the computer unit of the data processing apparatus is adapted to execute the steps of the method of any of claims 1 to 13.

15. The system according to claim 14, characterised in that the data processing apparatus is configured to communicate with a positioning system via a communicationslink, wherein the positioning system is configured to input position data to the data processing apparatus.

16. The system according to claims 14 or 15, characterised in that the computer unit is configured to combine the image data from multiple cameras into a multi-dimensional mapping of the industrial site.

17. The system according to any one of claims 14 to 16, characterised in that the system further comprises- at least two radio communications devices configured to communicate with each other via a radio communications link, the radio communication between the radio communications devices being transmitted to the data processing apparatus, and / or- at least one microphone connected to the data processing apparatus, wherein the at least one microphone is configured to pick up audio signals from at least one position at the industrial site.

18. The system according to any one of claims 14 to 17, characterised in that the system further comprises a plurality of telemetry devices located within the industrial site and / or at least one meteorological unit or system, wherein the telemetry devices and / or the meteorological unit or system are configured to communicate with the data processing apparatus via one or more communications links.

19. The system according to any one of claims 14 to 18, characterised in that the industrial site is an installation or shipping vessel, a shipping terminal, an installation or construction site, a heavy loading or unloading area, or a production or pre-assembly factory or area.

20. A computer program comprising instructions which, when loaded and run on the system of any one of claims 14 to 19, causes the data processing apparatus to execute the method steps of any one of claims 1 to 13.

21. A computer-readable medium having stored thereon the computer program of claim 20.

Images