A risk identification and response method based on object detection and edge-cloud collaboration

Abstract

The present invention discloses a risk identification and response method based on object detection and edge-cloud collaboration. It includes: 1) The edge storage unit stores video data in real time; 2) The edge computing unit executes the object detection program; 3) When the object detection program discovers a risk source, determine the type of the risk source; the target tracking module focuses the camera on the risk source for continuous tracking; 4) The edge computing unit executes the risk assessment module program to obtain the risk level P of the risk source. 5) The edge computing unit executes the risk response module program, looks up the risk response table according to the values of the risk source and the risk level, and obtains the alarm audio file name Z; 6) The risk response module sends the audio file to the alarm unit; 7) Loop and execute steps 3) to 6) until the object detection module discovers that the risk source disappears; 8) The edge computing unit sends this risk event to the cloud; 9) The edge computing unit receives the risk level table and the risk response table sent by the cloud computing unit.

Description

Technical Field

[0001] This method relates to the fields of machine vision, edge-cloud collaboration, and production safety management, and is applicable to the intelligent management of risky production, operation, and operation sites in industries such as industry, energy, and construction. Compared with the existing information technologies for on-site production safety management, this method can more quickly lock in hazard sources, evaluate risk factors, and respond promptly through edge computing and edge-cloud collaboration; compared with the traditional supervision mode of video recognition and manual activation of safety response and disposal at a remote end, this method can not only perform fast and accurate integrated risk identification, evaluation, and disposal, and try to eliminate risks at the first time, but also optimize the calculation model in a timely manner based on edge-cloud collaboration to improve performance. Background Art

[0002] Safety is an eternal theme in the production process. With the development of society and the improvement of productivity, production capacity is continuously increasing, and operations are becoming more large-scale. It has become increasingly important to ensure the efficiency and safety of production operations through digital, information-based, and refined management. However, at present, most production, operation, and operation sites cannot conduct intelligent management of personnel, materials, and equipment in a real-time and efficient manner, and generally have the following problems:

[0003] 1) Traditional video surveillance systems are not used efficiently. Abnormal information must be obtained through online synchronous observation by personnel or playback query, and then analyzed and judged by personnel throughout the process. This not only requires a large amount of time and effort, but also cannot comprehensively and real-time control the on-site safety situation.

[0004] 2) It is difficult to monitor in real time the working positions, operation specifications, and movement trajectories of employees, vehicles, and operating devices within the space range of key production operation areas and key operating facilities; it is difficult to promptly identify and intervene in sudden foreign objects.

[0005] 3) It is difficult to judge in a timely manner and handle emergencies such as entering a dangerous area by mistake, non-compliant operations, and occurrence of dangerous situations.

[0006] With the development of machine vision technology and edge-cloud collaborative computing technology, new methods have emerged to solve the above problems. Summary of the Invention

[0007] In view of the foregoing problems, the present invention proposes a risk identification and response method based on object detection and edge-cloud collaboration, and this method has the following characteristics:

[0008] To achieve the above object, a risk identification and response method based on object detection and edge-cloud collaboration according to the present invention, the method includes the following steps:

[0009] 1) The edge storage unit stores in real time the video data collected by the video acquisition unit;

[0010] 2) After the edge computing unit reads the configuration file from the edge storage unit to complete algorithm initialization, it reads the video uploaded by the video acquisition unit from the edge storage unit in real time and executes the target detection program;

[0011] 3) When the target detection program discovers a risk source, it determines the type of the risk source; the target tracking module generates an instruction to drive the rotatable device of the video acquisition unit to rotate by calculating the coordinate change relationship of the risk source between different frames of the video, so as to focus the camera on the risk source for continuous tracking;

[0012] 4) The edge computing unit executes the code of the risk assessment module to obtain the risk level P of the risk source.

[0013] 5) The edge computing unit executes the code of the risk response module, searches the risk response table according to the values of the risk source and the risk level, and obtains the alarm audio file name Z;

[0014] 6) The risk response module reads the audio file with the file name Z from the storage unit and sends it to the alarm unit; at the same time, according to the different risk levels, it drives the alarm unit to play the audio file according to the corresponding alarm driving rules.

[0015] 7) Loop and execute steps 3) to 6) until the target detection module discovers that the risk source disappears;

[0016] 8) The edge computing unit sends the video segment of this risk event, as well as the occurrence time, end time, risk source type, and risk level to the cloud;

[0017] 9) The edge computing unit receives: After the knowledge management module of the cloud computing unit evaluates the risk event, the risk level table and risk response table maintained according to expert experience.

[0018] Further, the method further includes the following steps:

[0019] 10) The edge computing unit receives: The deep learning module of the cloud computing unit uses a machine vision algorithm based on a convolutional neural network to add the newly acquired labeled video images to the training set, regularly optimizes the target detection model parameters to form a configuration file, and stores the configuration file in the edge storage unit.

[0020] Further, the method further includes the following steps:

[0021] After the cloud receives the risk message, the event management module of the cloud computing unit numbers the video clips parsed from the message, generates a risk event record by combining the data such as the occurrence time, end time, risk source type, and risk level parsed with the video clip number, and appends it to the risk history event table; the video clips are stored in the cloud storage unit.

[0022] Furthermore, the risk level table is constructed based on expert knowledge, successively taking the scenario working conditions, risk source location, risk source movement trend, and risk source residence duration as conditions when the risk source appears in the monitoring scenario.

[0023] Furthermore, the risk assessment module program discriminates the scenario working conditions, risk source location, risk source movement trend, and risk source residence time in the video, and then searches for the risk level table file in the second storage module to obtain the risk level of the risk source.

[0024] This method is applicable to the intelligent management of risky production, operation, and operation sites in industries such as industry, energy, and construction. Compared with the existing information technologies for on-site production safety management, this method can lock the hazard sources, evaluate risk factors, and respond quickly through edge computing and edge-cloud collaboration; compared with the traditional supervision mode of video recognition and manual activation of safety response and disposal at a remote end, this method can not only perform fast and accurate integrated risk identification, assessment, and disposal, and try to eliminate risks in the first time, but also optimize the calculation model in a timely manner based on edge-cloud collaboration to improve performance. Description of the Drawings

[0025] Figure 1 It is the composition of a risk identification and response system based on object detection and edge-cloud collaboration;

[0026] Figure 2 It is a schematic diagram of the internal core modules of the edge computing unit;

[0027] Figure 3 It is a schematic diagram of the internal core modules of the cloud computing unit;

[0028] Figure 4 It is an instance of a constructed risk level table;

[0029] Figure 5 It is an instance of a constructed risk response table;

[0030] Figure 6 It is an explanation of the running steps of a risk identification and response method based on object detection and edge-cloud collaboration. Detailed Implementation Manner

[0031] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It can be understood that the specific embodiments described herein are only for explaining the invention and not for limiting the present invention. Additionally, it should be noted that for the convenience of description, only the parts related to the present invention rather than all the content are shown in the drawings.

[0032] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention.

[0033] The terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, the meaning of "plurality" is two or more.

[0034] In the description of the present invention, it should be noted that unless otherwise clearly defined and limited, the terms "mounted", "connected", and "coupled" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0035] As an embodiment, a risk identification and response method based on object detection and edge-cloud collaboration first constructs a system composed of a video acquisition unit, an edge storage unit, an edge computing unit, an alarm unit, a first communication unit, a cloud storage unit, a cloud computing unit, a second communication unit, etc., as Figure 1 shown. A typical deployment method of this system is as follows:

[0036] 1) The video acquisition unit is a video acquisition device installed on-site. The video acquisition device includes an optical camera, a rotatable component for supporting the camera, and accessory fixing, light supplementing, and protection components. The rotatable component can drive the camera to rotate omnidirectionally up, down, left, and right under signal drive.

[0037] 2) The edge storage unit, edge computing unit, and first communication unit are deployed on an edge computing host that belongs to the same subnet as the video acquisition device. The memory of the edge computing host corresponds to the edge storage unit, the CPU and GPU processors correspond to the edge computing unit, and the network card corresponds to the first communication unit.

[0038] The edge storage unit consists of a first storage module, a second storage module, and a third storage module. The first storage module receives the data collected by the video acquisition unit and stores it in a rolling manner in chronological order; the second storage module stores the target detection algorithm configuration file, risk level table file, and risk correspondence table file; the third storage module stores voice warning audio files, and the files are in audio formats such as wav, wma, or mp3. The first storage module uses a large-capacity high-speed TF card, and the second and third storage modules share one TF card.

[0039] The edge computing unit consists of core modules such as target detection, target tracking, risk assessment, and risk response, as Figure 2 shown. The target detection module performs real-time analysis on the video flowing into the edge storage unit to complete the capture and identification of possible risk sources in the picture; the target tracking module generates an instruction to drive the rotatable device of the video acquisition unit to rotate by calculating the coordinate change relationship of the risk source between different frames of the video to track the position movement of the risk source; the risk assessment module determines the risk level of the risk source; the risk response module outputs response measures to the risk event according to the risk source and risk level, including the warning file retrieved from the risk response table and a set of warning action rules agreed in the module code, and drives the warning unit to give an alarm according to the warning action rules.

[0040] The warning action rules stipulate the characteristics of the warning behavior according to different risk levels. Here, the characteristics can be the number of times the warning file is played repeatedly and the control of the volume.

[0041] The first communication unit is used to send and receive messages based on IP packets with the second communication unit, and its carrier can be a wired network card or a wireless network card.

[0042] 3) The warning unit is a sound-emitting device installed on-site and consists of an audio drive module and a speaker. When the risk response module of the edge computing unit outputs a voice warning audio file, the file is played through the audio drive module and the speaker.

[0043] 4) The cloud storage unit, cloud computing unit, and second communication unit are deployed in the cloud server. Specifically, the cloud can be a private cloud, a public cloud, or a hybrid cloud, and the cloud storage unit, cloud computing unit, and second communication unit can run on virtual machines or Docker containers.

[0044] The cloud storage unit consists of a fourth storage module and a fifth storage module. The fourth storage module stores the video data of risk events; the fifth storage module stores the structured data regarding risk events.

[0045] The cloud computing unit consists of core modules such as event management, knowledge management, and deep learning, as Figure 3 shown. The event management module is used to parse the packets received by the second communication unit, write the video clips of risk events into the fourth storage module, and manage and maintain the risk history event table located in the fifth storage module. The knowledge management module conducts post-evaluation and analysis of risk events, and maintains the risk level table and risk response table located in the memory of the cloud server according to expert experience. It can be considered that the risk level table and risk response table of the second storage module on the edge side are respectively snapshots of the risk level table and risk response table in the cloud server, and they have the same data structure.

[0046] The deep learning module adopts a machine vision algorithm based on a convolutional neural network, adds the newly acquired labeled video images to the training set, periodically optimizes the parameters of the object detection model, and periodically sends it to the edge side in the form of a configuration file through the second communication unit.

[0047] The second communication unit is deployed in the cloud, and is used to receive IP packets from the first communication unit, and send updates of the edge side object detection algorithm parameters, risk level table, and risk response table to the first communication unit.

[0048] The following combines Figure 4 and 5 to describe the preparatory work and key points of system initialization for the risk identification and risk response method using object detection and edge-cloud collaboration:

[0049] 1) List the set of risk sources S = {S i}, i ∈ [1, I]. S i is a certain risk source that needs to be controlled separately, which can be a person, an object, or a certain event. The positive integer I is the number of types of risk sources included in S.

[0050] 2) Collect a sufficient number of images of various risk sources S i and label them. In the offline state, use a machine vision algorithm based on a convolutional neural network to train the above images to obtain the object detection model parameters, and store them in the second storage module of the edge storage unit in the form of a configuration file. Optionally, the machine vision algorithm based on a convolutional neural network adopts the YOLO-v4 algorithm.

[0051] 3) Based on expert knowledge, in sequence, when a risk source S i, evaluate the risk level f(S = i|C = C of the risk source i under the combined conditions with the quantified scenario working condition C, risk source location D, risk source movement trend M, and risk source residence duration T j , D = D k , M = M a , T = T b ) and briefly record it as P i|j,k,a,b , where C j , D k , M a , T b are the available values after classifying the scenario working condition C, risk source location D, risk source movement trend M, and risk source residence time T according to the predetermined rules, and f is the risk assessment rule. For any risk source S i 's all P i|j,k,a,b constitute a risk level sub-table P i |(C, D, M, T). All risk level sub-tables are combined into a risk level table P and stored in the second storage module of the edge storage unit in the form of a configuration file.

[0052] Figure 4 is an example of the risk level table. For this example, the value of S is i ∈ [1, 7]; the value of C is j ∈ {1, 2, 3}, where 1 represents'stoppage', 2 represents'slow operation', and 3 represents 'fast operation'; the value of D is k ∈ {1, 2}, where 1 represents 'long distance' and 2 represents'short distance'; the value of M is a ∈ {1, 2, 3, 4}, where 1 represents'slow approach', 2 represents 'fast approach', 3 represents'stay', and 2 represents 'away'; the value of T is b ∈ {1, 2, 3}, where b = 1 when 0 < t ≤ 10s, b = 2 when 10 < t ≤ 30s, and b = 3 when t > 30s, and t is the duration of this state.

[0053] 4) Develop corresponding risk response methods for each value of the risk level sub-table P i . Use O to represent the set of risk response methods. For each newly added item of the risk response method in the set O, generate an audio file with a unique name by means of voice synthesis or manual recording respectively. It should be noted that for different combinations (i, j, k, a, b), the adopted risk disposal methods can be the same, that is, mapped to the same audio file.

[0054] 5) Record the name of the audio file corresponding to the risk disposal method when the risk source S i is evaluated at the risk level P i|j,k,a,b as Z i|j,k,a,b , and record the set of all audio file names as Z. S i , P i|j,k,a,b and Z i|j,k,a,bForm a triple, and the set of all triples {S, P, Z} forms a risk response table, which is stored in the second storage module. Figure 5 Is an instance of the risk response table.

[0055] 6) Allocate computing, storage, and bandwidth resources through virtualization technology in the private cloud, and run the cloud computing unit, cloud storage unit, and second communication unit in the virtual server.

[0056] 7) Establish a communication link between the first communication unit of the edge computing host and the second communication unit of the cloud through a wired or wireless IP network.

[0057] Next, further combine Figure 6 Describe the working steps of the risk identification and risk response method based on object detection and edge-cloud collaboration adopted in this embodiment during operation:

[0058] 1) The key operating steps of the video acquisition device, edge computing host, and alarm device are as follows (as Figure 6 shown):

[0059] E1) The video acquisition device collects video signals of the monitored scene, and the collected video data is stored in the first storage module of the edge storage unit.

[0060] E2) The first communication unit monitors the messages from the second communication unit in real time. If it receives a message about the update of the object detection model configuration file, it updates the configuration file in the second storage module according to the message content; if it receives a message about the update of the risk level table, it updates the risk level table file in the second storage module according to the message content; if it receives a message about the update of the risk response table, it updates the risk response table file in the second storage module according to the message content;

[0061] E3) After the edge computing unit reads the configuration file from the third storage module to complete algorithm initialization, it reads the video from the edge storage module in real time and executes the object detection module code.

[0062] E4) When the object detection program discovers the risk source S, it determines the risk source type i. The target tracking module starts the target tracking algorithm when it discovers the risk source S i and outputs an instruction to drive the rotatable device of the camera unit, focusing the camera on the risk source for continuous tracking.

[0063] E5) The edge computing unit executes the risk assessment module code. First, it discriminates the scene working condition C, risk source location D, risk source movement trend M, and risk source residence time T, and then searches the risk level table file in the second storage module to obtain the risk level P of the risk source.

[0064] E6) The edge computing unit executes the risk response module. After reading the risk response table from the second storage module, it retrieves the corresponding Z according to S and P, and then reads the audio file Z from the third storage module and sends it to the alarm unit. The risk response module drives the alarm device to play the audio file according to the alarm action rules agreed upon for the current risk level. In this example, it is stipulated by the program code that when the risk level P ≤ 2, the audio file is played once; when 2 < P < 5, the audio file is played continuously twice; when P ≥ 5, the audio file is played continuously three times, and the volume is increased by one level during the subsequent play compared to the previous play.

[0065] E7) Loop through E4 to E6 until the target detection module discovers that the risk source has disappeared.

[0066] E8) The edge computing host sends the video clip of this risk event, as well as the occurrence time, end time, risk source type, and risk level data, to the second communication unit in the cloud through the first communication unit.

[0067] 2) The key operating steps of the cloud server are as follows (as Figure 6 shown):

[0068] C1) The second communication unit receives the message from the first communication unit.

[0069] C2) The event management module of the cloud server parses the message received by the second communication unit, numbers the obtained video clip, and appends the risk event record containing fields such as occurrence time, end time, risk source type, risk level data, and video clip number to the risk history event table, and writes the video clip into the fourth storage module.

[0070] C3) The knowledge management module has a web - or APP - based human - machine interface, which provides the viewing and editing of the current risk level table and risk response table, and the viewing of risk history events. Experts access the human - machine interface to re - check the risk level P i obtained when evaluating the risk source S i|j,k,a,b in the monitoring scenario. When the expert confirms that P i|j,k,a,b needs to be adjusted, the adjusted data is entered in the human - machine interface and written into the risk level table maintained by the cloud server. If the expert believes that it is also necessary to synchronously update the risk response method O i|j,k,a,bi and the corresponding Z i|j,k,a,b , the adjusted data is entered in the human - machine interface and written into the risk response table maintained by the cloud server. Then, the second communication unit sends the updates of the risk level table and risk response table to the first communication unit on the edge side.

[0071] Optionally, the re - check of the risk level for the received risk events in the cloud can be carried out by sampling at a ratio α, where α ∈ (0,1].

[0072] Preferably, each write operation on the risk level table and the risk response table during the expert review process in the background of the knowledge management module is stored separately in the form of a log for auditing.

[0073] C4) The deep learning module adopts a machine vision algorithm based on a convolutional neural network, adds the newly acquired labeled video images to the training set, periodically optimizes the target detection model parameters, and sends them to the first communication unit on the edge side in the form of a configuration file through the second communication unit. Optionally, the machine vision algorithm based on the convolutional neural network is the YOLO-v4 algorithm.

[0074] In the description of this specification, specific features, structures, materials, or characteristics may be combined in a suitable manner in any one or more embodiments or examples.

[0075] As described above, it is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A risk identification and response method based on object detection and edge-cloud collaboration, characterized in that, The method described above includes the following steps: 1) The edge storage unit stores the video data collected by the video acquisition unit in real time; 2) After the edge computing unit reads the configuration file from the edge storage unit to complete the algorithm initialization, it reads the video uploaded by the video acquisition unit from the edge storage unit in real time and executes the target detection program; 3) When the target detection program discovers a risk source, it determines the type of the risk source; The target tracking module generates an instruction to drive the rotatable device of the video acquisition unit to rotate by calculating the coordinate change relationship of the risk source between different frames of the video, so as to focus the camera on the risk source for continuous tracking; 4) The edge computing unit executes the risk assessment module program to obtain the risk level P of the risk source; 5) The edge computing unit executes the risk response module program, looks up the risk response table according to the values of the risk source and the risk level, and obtains the alarm audio file name Z; 6) The risk response module reads the audio file with the file name Z from the storage unit and sends it to the alarm unit; at the same time, according to the different risk levels, it drives the alarm unit to play the audio file according to the corresponding alarm driving rules; 7) Repeat steps 3) to 6) until the target detection module discovers that the risk source disappears; 8) The edge computing unit sends the video segment of this risk event, as well as the occurrence time, end time, risk source type, and risk level to the cloud; 9) The edge computing unit receives: After the knowledge management module of the cloud computing unit evaluates the risk event, it maintains the risk level table and the risk response table according to expert experience.

2. The risk identification and response method based on object detection and edge-cloud collaboration according to claim 1, characterized in that, The method described above further includes the following steps: 10) The edge computing unit receives: The deep learning module of the cloud computing unit uses a machine vision algorithm based on a convolutional neural network to add the newly acquired labeled video images to the training set, periodically optimizes the target detection model parameters to form a configuration file, and updates the configuration file to the edge storage unit.

3. The risk identification and response method based on object detection and edge-cloud collaboration according to claim 1, characterized in that, The method described above further includes the following steps: After the cloud receives the risk message, the event management module of the cloud computing unit numbers the video segment parsed from the message, generates a risk event record by combining the parsed occurrence time, end time, risk source type, and risk level data with the video segment number, and appends it to the risk history event table; the video segment is stored in the cloud storage unit.

4. The risk identification and response method based on object detection and edge-cloud collaboration according to claim 1, characterized in that, The risk level table is based on expert knowledge and is constituted by taking the scene working condition, risk source location, risk source movement trend, and risk source residence duration as conditions when the risk source appears in the monitoring scene in sequence.

5. The risk identification and response method based on object detection and edge-cloud collaboration according to claim 1, characterized in that, The risk assessment module program discriminates the scene working condition, risk source location, risk source movement trend, and risk source residence time in the video, and then looks up the risk level table file in the second storage module to obtain the risk level of the risk source.

Images