An industrial production safety data management method and system based on visual detection

Abstract

The present invention discloses an industrial production safety data management method and system based on visual detection, belonging to the technical field of safety detection. The system includes a data acquisition module, a visual perception module, a safety management module, and a visualization warning module; the data acquisition module is used to collect images of the production area and task plans of automated equipment; the visual perception module is used to build a three-dimensional scene, generate equipment models according to the automated equipment and divide the operation space area, generate object models through personnel images, calculate the first coefficient of the object models and identify risk object models; the safety management module marks the automated equipment with overlapping space areas, establishes a prediction space area for the marked automated equipment and calculates the second coefficient, and obtains the adjusted equipment according to the second coefficient and adjusts its task plan; the visualization warning module is used to display the object models and operation space areas in the three-dimensional scene, and automatically warns the staff when a safety hazard occurs.

Description

Technical Field

[0001] The present invention relates to the technical field of safety detection, and specifically to an industrial production safety data management method and system based on vision detection. Background Art

[0002] With the development of technology, robots and automated equipment are increasingly widely used in factories, covering multiple links such as manufacturing, assembly, and logistics. The introduction of automated equipment has significantly improved production efficiency and product quality. However, with the improvement of the automation level, production safety has increasingly become an important topic in safety management.

[0003] At present, in terms of safety management between equipment and personnel, visual detection algorithms are usually used to analyze images, establish a trigger range or directly perform collision detection to achieve safety management. This method has some problems. On the one hand, the setting of the trigger range uses undifferentiated detection. This method cannot distinguish between conscious active behaviors and unconscious passive behaviors of personnel. Simply making mechanical simple conditional judgments and triggers will lead to low efficiency and lack of intelligence in the coordinated work between equipment and people. On the other hand, the setting of collision detection is a post-remedial method. Production safety problems often occur at the moment of collision. It is impossible to evaluate the possible results of a collision before the collision, and the remedial measures taken after the collision cannot prevent the occurrence of safety accidents, still cannot reduce the risks in the production process, and cannot truly achieve safe production. Therefore, at present, a more efficient and intelligent production safety management technical solution is needed to solve the above problems. Summary of the Invention

[0004] The purpose of the present invention is to provide an industrial production safety data management method and system based on vision detection to solve the problems raised in the above background art.

[0005] To solve the above technical problems, the present invention provides an industrial production safety data management method based on vision detection, including the following steps:

[0006] S100. Collect the task plans of all automated equipment in the production area, collect images of the automated equipment from different angles through a camera, build a three-dimensional scene, and generate a device model for each automated equipment.

[0007] S200. Divide the operation space area in the three-dimensional scene according to the device model, generate an object model through the personnel image, analyze the positional relationship between the object model and the operation space area, calculate the first coefficient, and identify the risk object model.

[0008] S300. For the automated equipment with a spatial overlap area with the marked and risk object model, establish a prediction space area for the marked automated equipment and calculate the second coefficient, and obtain the adjusted equipment according to the second coefficient and adjust its task plan.

[0009] S400. Display the object model and the operation space area in the three-dimensional scene through a visualization interface, and automatically give an early warning to the staff when a potential safety hazard occurs.

[0010] In S100, the production area refers to the operation area for industrial production. The automated equipment refers to the mechanical equipment that completes specific process tasks in an automated manner. The task plan refers to the task being executed by the automated equipment, including the codes and execution durations of each action. An action refers to the basic motion unit of the automated equipment, and the execution duration refers to the time required for the automated equipment to execute the action. The automated equipment sequentially executes each action to complete the task plan. The three-dimensional scene refers to the three-dimensional space used to simulate the volume sizes and relative positions of all automated equipment in the production area, and the equipment model refers to the three-dimensional drawing used to simulate the automated equipment. Image acquisition is simultaneously performed through cameras installed at various angles in the production area, the common feature points in adjacent images are extracted for matching and the overlapping areas with each other and the same automated equipment are marked, and the multi-view three-dimensional reconstruction algorithm is used to combine the images from multiple views to build the three-dimensional scene, and the equipment models are respectively generated for each automated equipment in the three-dimensional scene.

[0011] In S200, the specific steps are as follows:

[0012] S201. Establish an action set for each automated equipment, and put the code DM of the action being executed in the task plan now into the corresponding action set. Set the prediction duration U z , when the sum of the execution durations of the actions corresponding to all the codes in the action set is less than U z , continue to put the next code after DM now into the corresponding action set in the execution order until the sum of the execution durations of the actions corresponding to all the codes in each action set is greater than or equal to U z .

[0013] S202. Analyze the spatial position change of the automated equipment caused by each action in the action set, simulate the three-dimensional space required for the equipment model to occupy when executing the actions in the corresponding action set in the three-dimensional scene, and combine the three-dimensional spaces of all the actions in the action set as the operation space area of the corresponding automated equipment. The YOLOv5 human detection algorithm is used to analyze the image information captured by each camera in real time. When a human object is detected in the production area, an object model is generated at the corresponding position in the three-dimensional scene through multi-view images, and the dynamic information of the human object in reality is mapped to the object model in real time.

[0014] Each action set can only generate one workspace area. The three-dimensional spaces occupied by all actions in the action set are combined to obtain the workspace area. If an action is in a static state during the duration, the corresponding three-dimensional space is the three-dimensional area occupied by the equipment model when maintaining this action. If an action is in a moving state during the duration, the corresponding three-dimensional space is the three-dimensional area passed by the equipment model when executing this action.

[0015] The generation of the object model is for virtual mapping in the real environment. According to the size and position of each person in the production area, the object model is generated in the three-dimensional scene, and the behavior actions and position changes of the person are obtained in real time and synchronously mapped to the three-dimensional scene for data analysis.

[0016] S203. Identify the spatial overlap area where the object model in the three-dimensional scene intersects with the workspace area, and mark the object models with spatial overlap areas. Use the key point detection algorithm to identify and mark different parts of the object model and label the head key points. Combine the head pose detection algorithm to analyze the spatial position changes of the head key points to obtain the line-of-sight direction S of the marked object model. Using the spatial position coordinates (X h , Y h , Z h ) of the center of the head of the marked object model as the starting point and S as the direction to establish a vector

[0017] The spatial overlap area represents the predicted collision area between the person and the equipment. The object model is a real-time mapped model, and the workspace area includes the three-dimensional area currently occupied and the three-dimensional area predicted to be occupied in the future.

[0018] S204. Set the angle threshold E, and analyze in real time the angle between the vector and the spatial position coordinates (X k , Y k , Z k ) of the center point of the corresponding spatial overlap area. When the angle is less than E, record the duration time time when each angle remains unchanged and the average spatial distance between (X h , Y h , Z h ) and (X k , Y k , Z k ) during this duration. Substitute into the formula to calculate the first coefficient of each marked object model. Set the first coefficient threshold C, and regard the marked object models with the first coefficient less than C as risk object models. The first coefficient calculation formula is as follows:

[0019]

[0020] Wherein, XS first is the first coefficient, is the duration of the g-th operation space area at the i-th time. is the (X h , Y h , Z h ) and (X k , Y k , Z k ) between the average spatial distance. is the angle of the g-th operation space area at the i-th time.

[0021] The first coefficient represents the degree of attention of personnel to automated equipment. When the automated equipment may collide with the current position of the personnel, the smaller the angle between the line of sight of the personnel and the center point of the collision area, the longer the duration, and the shorter the distance, the higher the degree of attention of the personnel to the potential collision, and the lower the probability of an accident, the smaller the first coefficient.

[0022] In S300, the specific steps are as follows:

[0023] S301. Obtain the volume VO of the spatial overlapping area between the risk object model and each operation space area sum , and calculate the average value VO ave of all the spatial overlapping area volumes corresponding to each risk object model. Mark the automated equipment corresponding to the operation space area with a spatial overlapping area. Set the prediction duration set {U1, U2,..., U z}, and generate an action set for each element in the prediction duration set according to the steps in S201. After combining the three-dimensional spaces of all the actions in each action set, use them as the prediction space area of the corresponding marked automated equipment at {U1, U2,..., U z}.

[0024] The number of prediction space areas is z. Each prediction duration corresponds to an action set, each action set corresponds to a prediction space area, and the volume of the prediction space area is proportional to the prediction duration.

[0025] S302. Re-identify the volume VC of the spatial overlapping area where the risk object model and each prediction space area under the corresponding marked automated equipment in the three-dimensional scene intersect r , and calculate the average value VC ave of these spatial overlapping area volumes. Analyze the spatial distance DI n between the spatial position coordinates (X n , Y n , Z r ) of the head center of the risk object model and the spatial position coordinates of the center point of the spatial overlapping area, calculate the average value DI of these spatial distances ave . Substitute into the formula to calculate the second coefficient of each marked automated device, set the second coefficient threshold B, and regard the marked automated device with the second coefficient greater than B as the adjustment device. The formula for calculating the second coefficient is as follows:

[0026]

[0027] In the formula, XS second is the second coefficient, and α is a constant.

[0028] The second coefficient represents the degree of danger of the automated device. When the volume of the spatial overlap area of the predicted spatial area of the automated device becomes larger over time and the spatial distance from the risk object model becomes larger over time, the higher the second coefficient, the more automated adjustment is required to avoid collisions between the automated device and personnel.

[0029] S303. Feed back the adjustment device information to the operation center and control the adjustment device to adjust the task plan, so that there is no intersecting spatial overlap area between the operation space area generated by the new task plan and any object model in the three-dimensional scene.

[0030] The adjustment task plan is automatically executed by a program preset for the automated device. Each time the automated device adjusts the task plan, it automatically feeds back to the operation center, and the operation center can intervene or not according to the feedback information.

[0031] In S400, through the visualization interface of the operation center, display the positional relationship between each object model and the operation space area in the three-dimensional scene in the form of dynamic images, highlight the adjustment devices with potential safety hazards and the new task plan after automatic adjustment, and give an early warning to the staff to judge whether to intervene.

[0032] An industrial production safety data management system based on visual detection includes a data acquisition module, a visual perception module, a safety management module, and a visualization early warning module.

[0033] The data acquisition module is used to collect images of the production area and the task plans of automated devices. The visual perception module is used to build a three-dimensional scene, generate device models according to automated devices and divide the operation space area, generate object models through personnel images, calculate the first coefficient of the object models and identify risk object models. The safety management module marks the automated devices with spatial overlap areas, establishes a predicted spatial area for the marked automated devices and calculates the second coefficient, obtains the adjustment devices according to the second coefficient and adjusts their task plans. The visualization early warning module is used to display the object models and the operation space area in the three-dimensional scene, and automatically give an early warning to the staff when potential safety hazards occur.

[0034] The data acquisition module includes an image information acquisition unit and a task plan acquisition unit.

[0035] The image information acquisition unit acquires images through cameras installed at different angles in the production area, where the production area refers to the work area for industrial production.

[0036] The task plan acquisition unit is used to acquire the task plans of each automated device. An automated device refers to a mechanical device that completes specific process tasks in an automated manner. A task plan refers to the task being executed by the automated device, including the codes and execution durations of each action. An action refers to the basic motion unit of the automated device, and the execution duration refers to the time required for the automated device to execute the action. The automated device sequentially executes each action to complete the task plan.

[0037] The visual perception module includes a region division unit and a behavior analysis unit.

[0038] The region division unit is used to build a three-dimensional scene and divide regions.

[0039] First, acquire images from all angles of the production area, extract the common feature points in adjacent images for matching, mark the overlapping regions and the same automated devices, combine the images from multiple perspectives to build a three-dimensional scene, and generate device models for each automated device in the three-dimensional scene.

[0040] Second, establish an action set for each automated device, set the prediction duration U z , obtain the code DM of the action being executed in the task plan now , and starting from DM now , sequentially put the subsequent codes into the corresponding action sets in order until the sum of the execution durations of the actions corresponding to all the codes in each action set is greater than or equal to U z .

[0041] Finally, simulate the three-dimensional space occupied by the device model when executing the actions in the corresponding action set in the three-dimensional scene, and combine the three-dimensional spaces of all the actions in the action set as the operation space area of the corresponding automated device. Analyze the images captured by each camera in real time, generate an object model at the corresponding position in the three-dimensional scene when a human object is detected in the production area, and map the dynamic information of the real human object to the object model in real time.

[0042] The behavior analysis unit is used to calculate the first coefficient of the object model and identify the risk object model.

[0043] First, identify the spatial overlap area where there is an intersection between the object model and the working space area in the three-dimensional scene, and mark the object models with spatial overlap areas. Use the key point detection algorithm combined with the head pose detection algorithm to analyze the spatial position changes of the key points on the head of the marked object model to obtain the line-of-sight direction S. Take the spatial position coordinates (X h , Y h , Z h ) of the center of the head of the marked object model as the starting point and establish a vector in the direction of S

[0044] Secondly, set the included angle threshold E, and analyze in real time the included angle between the vector and the spatial position coordinates (X k , Y k , Z k ) of the center point of each corresponding spatial overlap area. When the included angle is less than E, record the continuous duration time each time the included angle remains unchanged and the average value of the spatial distance between (X h , Y h , Z h ) and (X k , Y k , Z k ) during this duration.

[0045] Finally, calculate the first coefficient of each marked object model through the formula . Set the first coefficient threshold C, and regard the marked object models with the first coefficient less than C as risk object models. Among them, XS first is the first coefficient, is the continuous duration of the g-th working space area at the i-th time; is the average value of the spatial distance between (X h , Y h , Z h ) and (X k , Y k , Z k ) of the g-th working space area at the i-th time; is the included angle of the g-th working space area at the i-th time.

[0046] The safety management module includes a safety prediction unit and an emergency management unit.

[0047] The safety prediction unit is used to mark the automated equipment to establish a prediction space area and calculate the second coefficient.

[0048] First, obtain the volume VO sum of the spatial overlap area between the risk object model and each working space area, and calculate the average value VO ave of the volumes of all spatial overlap areas corresponding to each risk object model respectively.; Mark the automated equipment corresponding to the operation space area with overlapping spatial areas.

[0049] Secondly, set the prediction duration set {U1, U2,..., U z}, generate an action set for each element in the prediction duration set, and combine the three-dimensional spaces of all actions in each action set respectively as the predicted space area of the corresponding marked automated equipment at {U1, U2,..., U z}.

[0050] Then, identify again the volume VC of the spatial overlapping area where there is an intersection between the risk object model in the three-dimensional scene and each predicted space area under the corresponding marked automated equipment r , calculate the average value VC of these spatial overlapping area volumes ave ; Analyze the spatial position coordinates (X n , Y n , Z n ) of the head center of the risk object model and the spatial position coordinates of the center point of the spatial overlapping area, and calculate the spatial distance DI r between them, and calculate the average value DI of these spatial distances ave .

[0051] Finally, substitute into the formula to calculate the second coefficient of each marked automated equipment, set the second coefficient threshold B, and regard the marked automated equipment with the second coefficient greater than B as the adjustment equipment; where XS second is the second coefficient and α is a constant.

[0052] The emergency management unit feeds back the adjustment equipment information to the operation center and controls the adjustment equipment to adjust the task plan, so that the operation space area generated by the new task plan does not have an intersecting spatial overlapping area with any object model in the three-dimensional scene.

[0053] The visualization warning module displays the positional relationship between each object model and the operation space area in the three-dimensional scene through the visualization interface of the operation center, and displays the adjustment equipment and the adjusted new task plan, and warns the staff.

[0054] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0055] 1. Intelligent screening and prediction: This application first builds a three-dimensional scene to generate equipment models and object models, and then divides the operation space area according to the equipment model and the task plan to obtain spatial overlapping areas. By analyzing the angles, distances, and durations between the object model and each spatial overlapping area, the first coefficient is calculated, so as to screen out the personnel who need attention. It is more user-friendly and intelligent than the mechanical range-triggered screening method of the prior art.

[0056] 2. Efficient contrast detection: In this application, the risk object model and the marked automation equipment in the three-dimensional scene are first identified, and then several prediction space regions are established for each marked automation equipment. The volume change and spatial distance change of the spatial overlap region where the prediction space region intersects with the risk object model are analyzed. The second coefficient of each marked automation equipment is calculated, and the contrast detection between different marked automation equipment is realized according to the second coefficient. Compared with the collision-triggered detection in the prior art, it has the advantage of early prediction and is more efficient and reliable in estimating the risk level.

[0057] In summary, compared with the traditional technology, the present invention has the advantages of intelligent screening prediction and efficient contrast detection, and can improve the efficiency of production safety management. Description of the Drawings

[0058] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, and do not constitute a limitation to the present invention. In the drawings:

[0059] Figure 1 is a schematic flowchart of a method for industrial production safety data management based on vision detection according to the present invention;

[0060] Figure 2 is a schematic structural diagram of a system for industrial production safety data management based on vision detection according to the present invention. Detailed Embodiments

[0061] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0062] Please refer to Figure 1 , the present invention provides a method for industrial production safety data management based on vision detection, including the following steps:

[0063] S100. Collect the task plans of all automation equipment in the production area, collect images of the automation equipment from different angles through a camera, build a three-dimensional scene, and generate an equipment model for each automation equipment.

[0064] S200. Divide the operation space region in the three-dimensional scene according to the equipment model, generate an object model through the personnel image, analyze the positional relationship between the object model and the operation space region, calculate the first coefficient, and identify the risk object model.

[0065] S300. For the automated equipment whose marked area overlaps with the risk object model, establish a prediction space area for the marked automated equipment, calculate the second coefficient, obtain the adjusted equipment according to the second coefficient, and adjust its task plan.

[0066] S400. Display the object model and the operation space area in the three-dimensional scene through a visualization interface, and automatically give an early warning to the staff when a potential safety hazard occurs.

[0067] In S100, the production area refers to the working area for industrial production. Automated equipment refers to mechanical equipment that completes specific process tasks in an automated manner. The task plan refers to the task being executed by the automated equipment, including the codes and execution durations of each action. An action refers to the basic movement unit of the automated equipment, and the execution duration refers to the time required for the automated equipment to execute the action. The automated equipment sequentially executes each action to complete the task plan. The three-dimensional scene refers to a three-dimensional space used to simulate the volume sizes and relative positions of all automated equipment in the production area, and the equipment model refers to a three-dimensional drawing used to simulate the automated equipment. Images are collected simultaneously by cameras installed at various angles in the production area, common feature points in adjacent images are extracted for matching, and overlapping areas and the same automated equipment are marked. The multi-view three-dimensional reconstruction algorithm is used to combine images from multiple perspectives to build a three-dimensional scene, and equipment models are generated for each automated equipment in the three-dimensional scene.

[0068] In S200, the specific steps are as follows:

[0069] S201. Establish an action set for each automated equipment, and put the code DM of the action being executed in the task plan now into the corresponding action set. Set the prediction duration U z . When the sum of the execution durations of the actions corresponding to all the codes in the action set is less than U z , continue to put the next code after DM now into the corresponding action set in the execution order until the sum of the execution durations of the actions corresponding to all the codes in each action set is greater than or equal to U z .

[0070] S202. Analyze the spatial position changes of the automated equipment caused by each action in the action set, simulate the three-dimensional space required for the equipment model to occupy when executing the actions in the corresponding action set in the three-dimensional scene, and combine the three-dimensional spaces of all the actions in the action set as the operation space area of the corresponding automated equipment. Use the YOLOv5 human detection algorithm to analyze the image information captured by each camera in real time. When a human object is detected in the production area, generate an object model at the corresponding position in the three-dimensional scene by taking multi-view images, and map the dynamic information of the real human object to the object model in real time.

[0071] Each action set can only generate one workspace area. The three-dimensional space occupied by all actions in the action set is combined to obtain the workspace area. If an action is in a static state during the duration, the corresponding three-dimensional space is the three-dimensional area occupied by the equipment model when maintaining this action. If an action is in a moving state during the duration, the corresponding three-dimensional space is the three-dimensional area passed by the equipment model when executing this action.

[0072] The generation of the object model is for virtual mapping in the real environment. According to the size and position of each person in the production area, the object model is generated in the three-dimensional scene, and the behavior actions and position changes of the person are obtained in real time and synchronously mapped to the three-dimensional scene for data analysis.

[0073] S203. Identify the spatial overlap area where the object model in the three-dimensional scene intersects with the workspace area, and mark the object models with spatial overlap areas. Use the key point detection algorithm to identify different parts of the marked object model and label the head key points. Combine the head pose detection algorithm to analyze the spatial position changes of the head key points to obtain the line-of-sight direction S of the marked object model. Use the spatial position coordinates (X h , Y h , Z h ) of the center of the head of the marked object model as the starting point and S as the direction to establish a vector

[0074] The spatial overlap area represents the predicted collision area between the person and the equipment. The object model is a real-time mapped model, and the workspace area includes the three-dimensional area currently occupied and the three-dimensional area predicted to be occupied in the future.

[0075] S204. Set the angle threshold E, and analyze the vector and the spatial position coordinates (X k , Y k , Z k ) of the center point of each corresponding spatial overlap area in real time. When the angle is less than E, record the continuous duration time when each angle remains unchanged and the average spatial distance between (X h , Y h , Z h ) and (X k , Y k , Z k ) during this duration. Substitute into the formula to calculate the first coefficient of each marked object model. Set the first coefficient threshold C, and use the marked object models with the first coefficient less than C as the risk object models. The first coefficient calculation formula is as follows:

[0076]

[0077] Wherein, XS first is the first coefficient, is the duration of the g-th operation space area at the i-th time. is the (X h , Y h , Z h ) and (X k , Y k , Z k ) of the g-th operation space area at the i-th time, and the average value of the spatial distances therebetween. is the included angle of the g-th operation space area at the i-th time.

[0078] The first coefficient represents the degree of attention of personnel to automated equipment. When there is a possibility of collision between the automated equipment and the current position of the personnel, the smaller the included angle between the line of sight of the personnel and the center point of the collision area, the longer the duration, and the shorter the distance, the higher the degree of attention of the personnel to the potential collision, and the lower the possibility of an accident, the smaller the first coefficient.

[0079] In S300, the specific steps are as follows:

[0080] S301. Obtain the volume VO of the spatial overlapping area between the risk object model and each operation space area sum , and calculate the average value VO ave of all the volumes of the spatial overlapping areas corresponding to each risk object model. Mark the automated equipment corresponding to the operation space area with a spatial overlapping area. Set the prediction duration set {U1, U2,..., U z}, and generate an action set for each element in the prediction duration set according to the steps in S201. After combining the three-dimensional spaces of all the actions in each action set, use it as the predicted space area of the corresponding marked automated equipment at {U1, U2,..., U z}.

[0081] The number of predicted space areas is z. Each prediction duration corresponds to an action set, and each action set corresponds to a predicted space area. The volume of the predicted space area is proportional to the prediction duration.

[0082] S302. Identify again the volume VC of the spatial overlapping area where there is an intersection between the risk object model and each predicted space area under the corresponding marked automated equipment in the three-dimensional scene r , and calculate the average value VC ave of these volumes of the spatial overlapping areas. Analyze the spatial distance DI n between the spatial position coordinates (X n , Y n , Z r ) of the center of the head of the risk object model and the spatial position coordinates of the center point of the spatial overlapping area, calculate the average value DI of these spatial distances ave . Substitute into the formula to calculate the second coefficient of each marked automated device, set the second coefficient threshold B, and regard the marked automated device with the second coefficient greater than B as the adjustment device. The formula for calculating the second coefficient is as follows:

[0083]

[0084] In the formula, XS second is the second coefficient, and α is a constant.

[0085] The second coefficient represents the degree of danger of the automated device. When the volume of the spatial overlap area of the spatial area predicted by the automated device becomes larger over time and the spatial distance from the risk object model becomes larger over time, the higher the second coefficient, and the more automated adjustment is required to avoid collisions between the automated device and personnel.

[0086] S303. Feed back the adjustment device information to the operation center and control the adjustment device to adjust the task plan, so that there is no intersecting spatial overlap area between the operation space area generated by the new task plan and any object model in the three-dimensional scene.

[0087] The adjustment task plan is autonomously executed by a program preset for the automated device. Each time the automated device adjusts the task plan, it automatically feeds back to the operation center, and the operation center can intervene or not according to the feedback information.

[0088] In S400, through the visualization interface of the operation center, display the positional relationship between each object model and the operation space area in the three-dimensional scene in the form of dynamic images, highlight the adjustment devices with potential safety hazards and the new task plan after automatic adjustment, and give an early warning to the staff to judge whether to intervene.

[0089] Please refer to Figure 2 , the present invention provides an industrial production safety data management system based on visual detection, including a data acquisition module, a visual perception module, a safety management module, and a visualization warning module.

[0090] The data acquisition module is used to collect images of the production area and the task plans of automated devices. The visual perception module is used to build a three-dimensional scene, generate device models according to automated devices and divide the operation space area, generate object models through personnel images, calculate the first coefficient of the object models and identify risk object models. The safety management module marks the automated devices with spatial overlap areas, establishes a predicted spatial area for the marked automated devices and calculates the second coefficient, obtains the adjustment devices according to the second coefficient and adjusts their task plans. The visualization warning module is used to display the object models and the operation space area in the three-dimensional scene, and automatically give an early warning to the staff when potential safety hazards occur.

[0091] The data acquisition module includes an image information acquisition unit and a task plan acquisition unit.

[0092] The image information acquisition unit acquires images through cameras installed at different angles in the production area, where the production area refers to the working area for industrial production.

[0093] The task plan acquisition unit is used to acquire the task plans of each automated device. An automated device refers to a mechanical device that completes specific process tasks in an automated manner. A task plan refers to the task being executed by the automated device, including the codes and execution durations of each action. An action refers to the basic motion unit of the automated device, and the execution duration refers to the time required for the automated device to execute the action. The automated device sequentially executes each action to complete the task plan.

[0094] The visual perception module includes a region division unit and a behavior analysis unit.

[0095] The region division unit is used to build a three-dimensional scene and divide regions.

[0096] First, acquire images from various angles in the production area, extract common feature points in adjacent images for matching, mark the overlapping regions and the same automated devices, combine the images from multiple perspectives to build a three-dimensional scene, and generate device models for each automated device in the three-dimensional scene.

[0097] Second, establish an action set for each automated device, set the prediction duration U z , obtain the code DM of the action being executed in the task plan now , and sequentially put the subsequent codes into the corresponding action sets in order from DM now until the sum of the execution durations of the actions corresponding to all the codes in each action set is greater than or equal to U z .

[0098] Finally, simulate the three-dimensional space occupied by the device model when executing the actions in the corresponding action set in the three-dimensional scene, and combine the three-dimensional spaces of all the actions in the action set as the working space area of the corresponding automated device. Analyze the images captured by each camera in real time, generate an object model at the corresponding position in the three-dimensional scene when a human object is detected in the production area, and map the dynamic information of the real human object to the object model in real time.

[0099] The behavior analysis unit is used to calculate the first coefficient of the object model and identify the risk object model.

[0100] First, identify the spatial overlap regions where there are intersections between the object models and the operation space regions in the three-dimensional scene, and mark the object models with spatial overlap regions. Use the key-point detection algorithm combined with the head pose detection algorithm to analyze the spatial position changes of the key points on the heads of the marked object models to obtain the line-of-sight direction S. Take the spatial position coordinates (X h , Y h , Z h ) of the center of the head of the marked object model as the starting point and establish a vector in the direction of S

[0101] Secondly, set the included angle threshold E, and analyze in real time the included angle between the vector and the spatial position coordinates (X k , Y k , Z k ) of the center points of the corresponding spatial overlap regions. When the included angle is less than E, record the continuous duration time each time the included angle remains unchanged and the average spatial distance between (X h , Y h , Z h ) and (X k , Y k , Z k ) during this duration.

[0102] Finally, calculate the first coefficient of each marked object model through the formula . Set the first coefficient threshold C, and regard the marked object models with the first coefficient less than C as risk object models. Among them, XS first is the first coefficient, is the continuous duration of the g-th operation space region at the i-th time; is the average spatial distance between (X h , Y h , Z h ) and (X k , Y k , Z k ) of the g-th operation space region at the i-th time; is the included angle of the g-th operation space region at the i-th time.

[0103] The safety management module includes a safety prediction unit and an emergency management unit.

[0104] The safety prediction unit is used to mark the automated equipment to establish a prediction space region and calculate the second coefficient.

[0105] First, obtain the volume VO sum of the spatial overlap regions between the risk object models and each operation space region, and calculate the average value VO ave; Mark the automated equipment corresponding to the work space area with overlapping spatial areas.

[0106] Secondly, set the prediction duration set {U1, U2,..., U z}, generate an action set for each element in the prediction duration set, and combine the three-dimensional spaces of all actions in each action set respectively as the predicted space area of the corresponding marked automated equipment at {U1, U2,..., U z}.

[0107] Then, identify again the volume VC of the overlapping spatial area where there is an intersection between the risk object model in the three-dimensional scene and each predicted space area under the corresponding marked automated equipment r , calculate the average value VC of these overlapping spatial area volumes ave ; Analyze the spatial position coordinates (X n , Y n , Z n ) of the center of the risk object model's head and the spatial position coordinates of the center point of the overlapping spatial area, and calculate the average value DI of these spatial distances r , calculate the average value DI of these spatial distances ave .

[0108] Finally, substitute into the formula to calculate the second coefficient of each marked automated equipment, set the second coefficient threshold B, and regard the marked automated equipment with the second coefficient greater than B as the adjustment equipment; where XS second is the second coefficient and α is a constant.

[0109] The emergency management unit feeds back the adjustment equipment information to the operation center and controls the adjustment equipment to adjust the task plan, so that the work space area generated by the new task plan does not have an overlapping spatial area that intersects with any object model in the three-dimensional scene.

[0110] The visualization warning module displays the positional relationship between each object model in the three-dimensional scene and the work space area through the visualization interface of the operation center, and displays the adjustment equipment and the adjusted new task plan, and warns the staff.

[0111] Example 1:

[0112] Suppose there is an overlapping spatial area between the marked object model and two work space areas A1 and A2. When the included angle threshold is 90°, the following 5 observation records of the marked object model are collected:

[0113] Observation record 1: A1 spatial overlapping area; included angle 30°; duration 2s; average spatial distance 0.5m;

[0114] Observation Record 2: Overlapping area of A1 space; included angle 120°; duration 5 s; average spatial distance 0.3 m;

[0115] Observation Record 3: Overlapping area of A2 space; included angle 60°; duration 3 s; average spatial distance 0.2 m;

[0116] Observation Record 4: Overlapping area of A2 space; included angle 120°; duration 2 s; average spatial distance 0.3 m;

[0117] Observation Record 5: Overlapping area of A2 space; included angle 30°; duration 1 s; average spatial distance 0.2 m;

[0118] Substitute into the formula to calculate the first coefficient of the marked object model:

[0119]

[0120] Then the first coefficient of the marked object model is 0.55.

[0121] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device.

[0122] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An industrial production safety data management method based on visual detection, characterized in that: The method includes the following steps: S100. Collect the task plans of all automated devices in the production area, collect images of the automated devices from different angles through cameras, build a three-dimensional scene, and generate device models for each automated device; S200. Divide the operation space area according to the device models in the three-dimensional scene, generate an object model through the personnel image, analyze the positional relationship between the object model and the operation space area, calculate the first coefficient, and identify the risk object model; S300. Mark the automated devices with spatial overlapping areas with the risk object model, establish a prediction space area for the marked automated devices, calculate the second coefficient, obtain the adjustment devices according to the second coefficient, and adjust their task plans; S400. Display the object model and the operation space area in the three-dimensional scene through a visualization interface, and automatically give an early warning to the staff when a safety hazard occurs; In S200, the specific steps are as follows: S201. Establish an action set for each automated device, and place the code DM of the action being executed in the task plan now into the corresponding action set; set the prediction duration U z . When the sum of the execution durations of the actions corresponding to all the codes in the action set is less than U z , continue to place the next code after DM now into the corresponding action set in the execution order until the sum of the execution durations of the actions corresponding to all the codes in each action set is greater than or equal to U z ; S202. Analyze the spatial position changes of each automated device caused by each action in the action set, simulate the three-dimensional space required for the device model to execute the actions in the corresponding action set in the three-dimensional scene, and combine the three-dimensional spaces of all actions in the action set as the operation space area corresponding to the automated device; Use the YOLOv5 human detection algorithm to analyze the image information captured by each camera in real time. When a personnel object is detected in the production area, generate an object model at the corresponding position in the three-dimensional scene by taking multi-perspective images, and map the dynamic information of the personnel object in reality to the object model in real time; S203. Identify the spatial overlap area where there is an intersection between the object model and the operation space area in the three-dimensional scene, and mark the object model with a spatial overlap area; use the key point detection algorithm to identify different parts of the marked object model and label the head key points, and analyze the spatial position change of the head key points in combination with the head pose detection algorithm to obtain the line-of-sight direction S of the marked object model; establish a vector with the spatial position coordinates (X h , Y h , Z h ) of the center of the head of the marked object model as the starting point and S as the direction S204. Set an included angle threshold E and analyze the vector in real time and the spatial position coordinates (X k , Y k , Z k ) of the center points of the corresponding spatial overlapping regions. When the included angle is less than E, record the duration time time when the included angle remains unchanged each time and the average spatial distance between (X h , Y h , Z h ) and (X k , Y k , Z k ) during this duration; substitute into the formula to calculate the first coefficient of each marked object model; set a first coefficient threshold C, and regard the marked object model with the first coefficient less than C as a risk object model; the calculation formula of the first coefficient is as follows: where XS first is the first coefficient, is the duration of the g-th operation space area at the i-th time; is the average spatial distance between (X h , Y h , Z h ) and (X k , Y k , Z k ) in the g-th operation space area at the i-th time; is the included angle of the g-th operation space area at the i-th time.

2. The industrial production safety data management method based on visual detection according to claim 1, wherein: In S100, the production area refers to the operation area for industrial production; the automated device refers to the mechanical equipment that completes specific process tasks in an automated manner; the task plan refers to the task being executed by the automated device, including the codes and execution durations of each action. The action refers to the basic motion unit of the automated device, and the execution duration refers to the duration required for the automated device to execute the action. The automated device sequentially executes each action to complete the task plan; the three-dimensional scene refers to the three-dimensional space used to simulate the volume sizes and relative positions of all automated devices in the production area, and the device model refers to the three-dimensional drawing used to simulate the automated device; Collect images simultaneously through cameras installed at various angles in the production area, extract the common feature points in adjacent images for matching, mark the overlapping areas and the same automated device, and use the multi-perspective three-dimensional reconstruction algorithm to combine the images from multiple perspectives to build a three-dimensional scene, and generate device models for each automated device in the three-dimensional scene.

3. A method for managing industrial production safety data based on visual detection according to claim 1, characterized in that: In S300, the specific steps are as follows: S301. Obtain the volume VO of the spatial overlapping region between the risk object model and each operation space region sum , and calculate the average value VO of the volumes of all spatial overlapping regions corresponding to each risk object model respectively ave ; Mark the automated equipment corresponding to the operation space region with a spatial overlapping region; Set the prediction duration set {U1, U2,..., U z}, and generate an action set for each element in the prediction duration set according to the steps in S201. After combining the three-dimensional spaces of all actions in each action set respectively, use them as the predicted space regions of the corresponding marked automated equipment at {U1, U2,..., U z}; S302. Identify again the volume VC of the spatial overlapping region where the risk object model in the three-dimensional scene intersects with each prediction space region under the corresponding marking automation device r , and calculate the average value VC of the volumes of these spatial overlapping regions ave ; Analyze the spatial distance DI n between the spatial position coordinates (X n , Y n ) of the center of the head of the risk object model and the spatial position coordinates of the center point of the spatial overlapping region r , and calculate the average value DI of these spatial distances ave ; Substitute into the formula to calculate the second coefficient of each marked automated device, set the second coefficient threshold B, and regard the marked automated devices with the second coefficient greater than B as the adjustment devices; The calculation formula of the second coefficient is as follows: where XS second is the second coefficient and α is a constant; S303. Feed back the adjustment device information to the operation center and control the adjustment device to adjust the task plan so that the operation space area generated by the new task plan does not have an intersecting spatial overlapping area with any object model in the three-dimensional scene.

4. The industrial production safety data management method based on visual detection according to claim 3, wherein: In S400, through the visualization interface of the operation center, the positional relationship between each object model in the three-dimensional scene and the operation space area is displayed in the form of a dynamic image, highlighting the adjustment devices with potential safety hazards and the new task plan after automatic adjustment, and warning the staff to judge whether to intervene.

5. An industrial production safety data management system based on visual detection, characterized in that: The system includes a data acquisition module, a visual perception module, a safety management module, and a visualization warning module; The data acquisition module is used to acquire images of the production area and the task plans of automated equipment; The visual perception module is used to build a three-dimensional scene, generate equipment models according to automated equipment and divide the operation space area, generate object models through personnel images, calculate the first coefficient of the object models and identify risk object models; The safety management module marks automated equipment with overlapping space areas, establishes a prediction space area for the marked automated equipment and calculates the second coefficient, and obtains the adjustment equipment according to the second coefficient and adjusts its task plan; The visualization warning module is used to display object models and operation space areas in the three-dimensional scene, and automatically warn the staff when a safety hazard occurs; The visual perception module includes a region division unit and a behavior analysis unit; The region division unit is used to build a three-dimensional scene and divide regions; First, images of various angles of the production area are acquired, common feature points in adjacent images are extracted for matching and the overlapping areas and the same automated equipment are marked, the images from multiple perspectives are combined to build a three-dimensional scene, and equipment models are generated for each automated equipment in the three-dimensional scene; Secondly, establish an action set for each automated device and set the prediction duration U z , obtain the code DM of the action being executed in the task plan now , starting from DM now , sequentially put the subsequent codes into the corresponding action sets in order of execution until the sum of the execution durations of the actions corresponding to all the codes in each action set is greater than or equal to U z ; Finally, the three-dimensional space occupied by the equipment model when performing the actions in the corresponding action set is simulated in the three-dimensional scene, and the three-dimensional spaces of all actions in the action set are combined as the operation space area of the corresponding automated equipment; The images captured by each camera are analyzed in real time. When a personnel object is detected in the production area, an object model is generated at the corresponding position in the three-dimensional scene, and the dynamic information of the personnel object in reality is mapped to the object model in real time; The behavior analysis unit is used to calculate the first coefficient of the object models and identify risk object models; First, identify the spatial overlap region where there is an intersection between the object model and the operation space region in the three-dimensional scene, and mark the object models with spatial overlap regions; use the key point detection algorithm combined with the head pose detection algorithm to analyze the spatial position changes of the key points of the head of the marked object model to obtain the line-of-sight direction S; use the spatial position coordinates (X h , Y h , Z h ) of the center of the head of the marked object model as the starting point and S as the direction to establish a vector Secondly, set an angle threshold E and analyze the vector in real time and the spatial position coordinates (X k , Y k , Z k ) of the center points of the corresponding spatial overlapping regions; when the angle is less than E, record the duration time when the angle remains unchanged each time and the average spatial distance between (X h , Y h , Z h ) and (X k , Y k , Z k ) during this duration; Finally, through the formula calculate the first coefficient of each labeled object model; set the first coefficient threshold C, and regard the labeled object model with the first coefficient less than C as the risk object model; where, XS first is the first coefficient, is the duration of the g-th job space area at the i-th time; is the average spatial distance between (X h , Y h , Z h ) and (X k , Y k , Z k ) in the g-th job space area at the i-th time; is the included angle of the g-th job space area at the i-th time.

6. The industrial production safety data management system based on visual detection according to claim 5, wherein: The data acquisition module includes an image information acquisition unit and a task plan acquisition unit; The image information acquisition unit acquires images through cameras installed at different angles in the production area, and the production area refers to the operation area for industrial production; The task plan acquisition unit is used to acquire the task plans of each automated equipment; Automated equipment refers to mechanical equipment that completes specific process tasks in an automated manner; The task plan refers to the task being executed by the automated equipment, including the codes and execution durations of each action. An action refers to the basic motion unit of the automated equipment, and the execution duration refers to the time required for the automated equipment to execute the action. The automated equipment sequentially executes each action to complete the task plan.

7. An industrial production safety data management system based on visual detection according to claim 5, characterized in that: The safety management module includes a safety prediction unit and an emergency management unit; The safety prediction unit is used to mark automated equipment, establish a prediction space area and calculate the second coefficient; First, obtain the volume VO of the spatial overlap region between the risk object model and each working space area sum , and calculate the average value VO of the volumes of all spatial overlap regions corresponding to each risk object model respectively ave ; Mark the automated equipment corresponding to the working space area where there is a spatial overlap region Secondly, set the prediction duration set {U1, U2,..., U z}, generate an action set for each element in the prediction duration set, and combine the three-dimensional spaces of all actions in each action set respectively as the predicted space region corresponding to the marked automation device at {U1, U2,..., U z}; Then, identify again the volume VC of the spatial overlapping region where the risk object model in the three-dimensional scene intersects with each predicted spatial region under the corresponding marking automation device r , and calculate the average value VC of the volumes of these spatial overlapping regions ave ; Analyze the spatial distance DI n between the spatial position coordinates (X n , Y n ) of the head center of the risk object model and the spatial position coordinates of the center point of the spatial overlapping region r , and calculate the average value DI of these spatial distances ave ; Finally, substitute into the formula to calculate the second coefficient of each marked automation device, set the second coefficient threshold B, and take the marked automation devices with the second coefficient greater than B as adjustment devices; where XS second is the second coefficient, and α is a constant; The emergency management unit feeds back the adjusted device information to the operation center and controls the adjustment device to adjust the task plan, so that there is no overlapping spatial area between the operation space area generated by the new task plan and any object model in the three-dimensional scene.

8. An industrial production safety data management system based on visual detection according to claim 7, characterized in that: The visualization warning module displays the positional relationship between each object model and the operation space area in the three-dimensional scene through the visualization interface of the operation center, as well as displays the adjusted device and the new task plan after adjustment, and warns the staff.

Images