Industrial production safety data management method and system based on visual inspection

Abstract

The invention discloses an industrial production safety data management method and system based on visual detection, and belongs to the technical field of safety detection. The system comprises a data acquisition module, a visual perception module, a safety management module and a visual early warning module. The data acquisition module is used for acquiring an image of a production area and a task plan of automatic equipment; the visual perception module is used for building a three-dimensional scene, generating an equipment model according to automation equipment, dividing a work space area, generating an object model through a personnel image, calculating a first coefficient of the object model and identifying a risk object model; the safety management module marks the automation equipment with the space overlapping area, a prediction space area is established for the marked automation equipment, a second coefficient is calculated, adjustment equipment is obtained according to the second coefficient, and a task plan of the adjustment equipment is adjusted; and the visual early warning module is used for displaying an object model and a working space area in the three-dimensional scene, and automatically early warning workers when potential safety hazards occur.

Description

Technical Field

[0001] The present invention relates to the field of safety detection technology, and in particular to an industrial production safety data management method and system based on visual detection. Background Art

[0002] With the development of science and technology, robots and automation equipment are increasingly used in factories, covering multiple links such as manufacturing, assembly, and logistics. The introduction of automation equipment has significantly improved production efficiency and product quality. However, with the improvement of automation level, production safety has also become an important issue in safety management.

[0003] At present, in terms of safety management between equipment and personnel, visual detection algorithms are usually used to analyze images, establish trigger ranges or directly perform collision detection to achieve safety management. This method has some problems. On the one hand, the setting of the trigger range adopts indifferent detection, which cannot distinguish between conscious active behavior and unconscious passive behavior of personnel. It is just a mechanical simple conditional judgment trigger, which will cause the coordination between equipment and people to become inefficient and not intelligent enough. On the other hand, the setting of collision detection is a post-event compensation method, and production safety problems often occur at the moment of collision. Before the collision, it is impossible to evaluate the possible consequences of the collision. The compensatory measures after the collision cannot avoid the occurrence of safety accidents, and still cannot reduce the risks in the production process and cannot truly achieve safe production. Therefore, at this stage, a more efficient and intelligent production safety management technology solution is needed to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to provide an industrial production safety data management method and system based on visual inspection to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides an industrial production safety data management method based on visual inspection, comprising the following steps:

[0006] S100, collect the task plans of all automation equipment in the production area, collect images of the automation equipment at different angles through cameras, build a three-dimensional scene and generate a device model for each automation equipment.

[0007] S200, dividing the working space area according to the equipment model in the three-dimensional scene, generating an object model through the personnel image, analyzing the positional relationship between the object model and the working space area to calculate the first coefficient and identify the risk object model.

[0008] S300, marking the automation equipment that has a spatial overlap area with the risk object model, establishing a predicted spatial area for the marked automation equipment and calculating a second coefficient, obtaining an adjustment device based on the second coefficient and adjusting its task plan.

[0009] S400, display the object model and work space area in the three-dimensional scene through a visual interface, and automatically warn the staff when a safety hazard occurs.

[0010] In S100, the production area refers to the work area used for industrial production. Automation equipment refers to mechanical equipment that completes specific process tasks in an automated manner. Task plan refers to the task that the automation equipment is performing, including the code and execution time of each action. Action refers to the basic motion unit of the automation equipment. The execution time refers to the time required for the automation equipment to perform the action. The automation equipment executes each action in sequence to complete the task plan. A three-dimensional scene refers to a three-dimensional space used to simulate the volume size and relative position of all automation equipment in the production area. The equipment model refers to a three-dimensional diagram used to simulate the automation equipment. Through the cameras installed at various angles in the production area, image acquisition is performed simultaneously, and common feature points in adjacent images are extracted for matching and marking of repeated areas and the same automation equipment. A multi-view three-dimensional reconstruction algorithm is used to combine images from multiple perspectives to build a three-dimensional scene, and a device model is generated for each automation equipment in the three-dimensional scene.

[0011] In S200, the specific steps are as follows:

[0012] S201, establish an action set for each automation device, and DM the code of the action being executed in the task plan now Put it into the corresponding action set. Set the prediction duration U z , when the sum of the execution time of all the codes corresponding to the actions in the action set is less than U z When the DM now The next code is put into the corresponding action set until the sum of the execution time of all the code corresponding actions in each action set is greater than or equal to U z .

[0013] S202, analyze the spatial position change of the automation equipment caused by each action in the action set, simulate the three-dimensional space occupied by the equipment model when executing the action in the corresponding action set in the three-dimensional scene, and combine the three-dimensional spaces of all actions in the action set as the working space area of ​​the corresponding automation equipment. Use the YOLOv5 human body detection algorithm to analyze the image information captured by each camera in real time. When the presence of human objects is detected in the production area, an object model is generated at the corresponding position in the three-dimensional scene through multi-view images, and the dynamic information of the human objects in reality is mapped to the object model in real time.

[0014] Each action set can only generate one workspace area. The three-dimensional spaces occupied by all actions in the action set are combined to form the workspace area. If the action is static during the duration, the corresponding three-dimensional space is the three-dimensional area occupied by the device model when it maintains the action. If the action is in motion during the duration, the corresponding three-dimensional space is the three-dimensional area that the device model passes through when executing the action.

[0015] The generation of object models is a virtual mapping of the real environment. The object model is generated in the three-dimensional scene according to the size and position of each person in the production area. The behavior and position changes of the personnel are obtained in real time and synchronously mapped to the three-dimensional scene for data analysis.

[0016] S203, identify the spatial overlap area between the object model and the work space area in the three-dimensional scene, and mark the object model with the spatial overlap area. Use the key point detection algorithm to identify different parts of the marked object model and mark the key points of the head. Combined with the head posture detection algorithm, analyze the spatial position change of the head key points to obtain the sight direction S of the marked object model. The spatial position coordinates (X h ,Y h ,Z h ) as the starting point and S as the direction to establish a vector

[0017] The spatial overlapping area represents the expected collision area between personnel and equipment. The object model is a real-time mapped model. The working space area includes the currently occupied three-dimensional area and the three-dimensional area expected to be occupied in the future.

[0018] S204, setting angle threshold E, real-time analysis of vector The spatial position coordinates (X k ,Y k ,Z k ) between them. When the angle is less than E, record the duration time when the angle remains unchanged and the time (X h ,Y h ,Z h ) and (X k ,Y k ,Z k ). Substitute the formula to calculate the first coefficient of each marked object model. Set the first coefficient threshold C, and take the marked object model with a first coefficient less than C as the risk object model. The first coefficient calculation formula is as follows:

[0019]

[0020] In the formula, XS first is the first coefficient, is the duration of the g-th working space area at the i-th time. is the g-th operation space area in the i-th time (X h ,Y h ,Z h ) and (X k ,Y k ,Z k ) is the average spatial distance between them. is the angle of the g-th working space area at the i-th time.

[0021] The first coefficient indicates the degree of attention paid by personnel to the automated equipment. When the automated equipment is likely to collide with the personnel's current location, the smaller the angle between the personnel's line of sight and the center point of the collision zone, the longer the duration, and the shorter the distance, the more it indicates that the personnel are more concerned about the collision that has not occurred. The lower the possibility of an accident, the smaller the first coefficient.

[0022] In S300, the specific steps are as follows:

[0023] S301: Obtaining the spatial overlap volume VO of the risk object model and each operation space area sum , calculate the average volume VO of all spatial overlapping areas corresponding to each risk object model ave . Mark the automation equipment corresponding to the work space area where there is spatial overlap. Set the prediction duration set {U1,U2,...,U z}, generate an action set for each element in the predicted duration set according to the steps in S201, and combine the three-dimensional spaces of all actions in each action set as the corresponding marked automation equipment in {U1, U2, ..., U z} when the predicted space area.

[0024] The number of prediction space regions is z, each prediction duration corresponds to an action set, each action set corresponds to a prediction space region, and the volume of the prediction space region is proportional to the prediction duration.

[0025] S302: Re-identify the spatial overlap area volume VC between the risk object model in the three-dimensional scene and each predicted spatial area under the corresponding marking automation device r , calculate the average volume VC of these spatial overlapping areas ave . Analyze the spatial position coordinates of the head center of the risk object model (X n ,Y n ,Z n ) and the spatial distance DI between the spatial position coordinates of the center point of the spatial overlapping area r, calculate the average value DI of these spatial distances ave Substitute the formula to calculate the second coefficient of each marking automation device, set the second coefficient threshold B, and use the marking automation device with a second coefficient greater than B as the adjustment device. The second coefficient calculation formula is as follows:

[0026]

[0027] In the formula, XS second is the second coefficient and α is a constant.

[0028] The second coefficient indicates the degree of danger of the automated equipment. When the volume of the spatial overlap area of ​​the automated equipment predicted spatial area increases over time, and the spatial distance from the risk object model increases over time, the higher the second coefficient is, the more automated adjustments are needed to avoid collisions between automated equipment and personnel.

[0029] S303, feeding back the adjustment device information to the operation center and controlling the adjustment device to adjust the task plan, so that the work space area generated by the new task plan does not have an intersecting spatial overlapping area with any object model in the three-dimensional scene.

[0030] The adjustment task plan is autonomously executed by the program pre-set for the automation equipment. Each time the automation equipment adjusts the task plan, it automatically feeds back to the operation center. The operation center can intervene or not take any action based on the feedback information.

[0031] In S400, the positional relationship between each object model and the working space area in the three-dimensional scene is displayed in the form of dynamic images through the visual interface of the operation center, highlighting the adjustment equipment with safety hazards and the new task plan after automatic adjustment, and warning the staff to determine whether to intervene.

[0032] An industrial production safety data management system based on visual detection includes a data acquisition module, a visual perception module, a safety management module and a visual early warning module.

[0033] The data acquisition module is used to collect images of the production area and the task plan of the automation equipment. The visual perception module is used to build a three-dimensional scene, generate equipment models based on the automation equipment and divide the work space area, generate object models through personnel images, calculate the first coefficient of the object model and identify the risk object model. The safety management module marks the automation equipment with overlapping spatial areas, establishes a predicted spatial area for the marked automation equipment and calculates the second coefficient, obtains the adjustment equipment according to the second coefficient and adjusts its task plan. The visual warning module is used to display the object model and work space area in the three-dimensional scene, and automatically warns the staff when a safety hazard occurs.

[0034] The data acquisition module includes an image information acquisition unit and a task plan acquisition unit.

[0035] The image information acquisition unit acquires images through cameras installed at different angles in the production area. The production area refers to the operating area used for industrial production.

[0036] The task plan collection unit is used to collect the task plan of each automation equipment. Automation equipment refers to mechanical equipment that completes specific process tasks in an automated manner. The task plan refers to the task that the automation equipment is performing, including the code and execution time of each action. The action refers to the basic motion unit of the automation equipment, and the execution time refers to the time required for the automation equipment to perform the action. The automation equipment executes each action in sequence to complete the task plan.

[0037] The visual perception module includes a region division unit and a behavior analysis unit.

[0038] The area division unit is used to build a three-dimensional scene and divide it into areas.

[0039] First, collect images from various angles of the production area, extract common feature points from adjacent images for matching, and mark repeated areas and the same automation equipment. Combine images from multiple perspectives to build a three-dimensional scene, and generate a device model for each automation equipment in the three-dimensional scene.

[0040] Secondly, establish an action set for each automated device and set the prediction duration U z , get the code DM of the action being executed in the task plan now , in order of execution from DM now Start putting the following codes into the corresponding action sets one by one, until the sum of the execution time of all the codes corresponding to the actions in each action set is greater than or equal to U z .

[0041] Finally, the 3D space occupied by the equipment model when executing the corresponding action set is simulated in the 3D scene, and the 3D space of all actions in the action set is combined as the working space area of ​​the corresponding automation equipment. The images taken by each camera are analyzed in real time. When the presence of personnel objects in the production area is detected, an object model is generated at the corresponding position in the 3D scene, and the dynamic information of the personnel objects in reality is mapped to the object model in real time.

[0042] The behavior analysis unit is used to calculate a first coefficient of the object model and identify a risky object model.

[0043] First, the spatial overlap area between the object model and the work space in the three-dimensional scene is identified, and the object model with the spatial overlap area is marked. The key point detection algorithm is combined with the head posture detection algorithm to analyze the spatial position change of the key points of the marked object model head to obtain the line of sight direction S. The spatial position coordinates of the center of the marked object model head (X h ,Y h ,Z h ) as the starting point and S as the direction to establish a vector

[0044] Secondly, set the angle threshold E and analyze the vector in real time. The spatial position coordinates (X k ,Y k ,Z k ). When the angle is less than E, record the duration of each time when the angle remains unchanged and the time (X h ,Y h ,Z h ) and (X k ,Y k ,Z k ) is the average spatial distance between them.

[0045] Finally, through the formula Calculate the first coefficient of each marked object model. Set the first coefficient threshold C, and take the marked object model with a first coefficient less than C as the risk object model. first is the first coefficient, is the duration of the g-th operating space area at the i-th time; is the g-th operation space area in the i-th time (X h ,Y h ,Z h ) and (X k ,Y k ,Z k ) between the average spatial distance; is the angle of the g-th working space area at the i-th time.

[0046] The safety management module includes a safety prediction unit and an emergency management unit.

[0047] The safety prediction unit is used to mark the automation equipment to establish a prediction space area and calculate the second coefficient.

[0048] First, obtain the spatial overlap volume VO of the risk object model and each operation space area sum , calculate the average volume VO of all spatial overlapping areas corresponding to each risk object model ave; Mark the automation equipment corresponding to the working space area where there is spatial overlap.

[0049] Secondly, set the prediction duration set {U1,U2,...,U z}, generate an action set for each element in the prediction duration set, and combine the three-dimensional spaces of all actions in each action set as the corresponding labeled automation equipment in {U1,U2,...,U z} when the predicted space area.

[0050] Then, the spatial overlap area volume VC between the risk object model in the three-dimensional scene and each predicted spatial area under the corresponding marking automation equipment is identified again r , calculate the average volume VC of these spatial overlapping areas ave ; Analyze the spatial position coordinates of the head center of the risk object model (X n ,Y n ,Z n ) and the spatial distance DI between the spatial position coordinates of the center point of the spatial overlapping area r , calculate the average value DI of these spatial distances ave .

[0051] Finally, substitute into the formula , calculate the second coefficient of each marking automation device, set the second coefficient threshold B, and use the marking automation device with a second coefficient greater than B as the adjustment device; wherein XS second is the second coefficient and α is a constant.

[0052] The emergency management unit will adjust the equipment information to the operation center and control the equipment to adjust the task plan so that the work space area generated by the new task plan does not have any intersecting spatial overlapping area with any object model in the three-dimensional scene.

[0053] The visual warning module displays the positional relationship between each object model and the work space area in the three-dimensional scene through the visual interface of the operation center, displays the adjusted equipment and the adjusted new task plan, and warns the staff.

[0054] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0055] 1. Intelligent screening prediction: This application first builds a three-dimensional scene to generate a device model and an object model, and then divides the work space area according to the device model and the task plan to obtain the spatial overlapping area. The first coefficient is calculated by analyzing the angle, distance and duration between the object model and each spatial overlapping area, so as to screen out the personnel who need attention. Compared with the mechanical range trigger screening method of the prior art, it is more humane and intelligent.

[0056] 2. Efficient comparative detection: This application first identifies the risk object model and marking automation equipment in the three-dimensional scene, and then establishes several predicted spatial areas for each marking automation equipment. Analyze the volume changes and spatial distance changes of the spatial overlapping areas where the predicted spatial area intersects with the risk object model. Calculate the second coefficient of each marking automation equipment, and implement comparative detection between different marking automation equipment based on the second coefficient. Compared with the collision-triggered detection in the prior art, it has the advantage of early prediction, and the estimation of the degree of risk is more efficient and reliable.

[0057] In summary, compared with traditional technologies, the present invention has the advantages of intelligent screening prediction and efficient comparative detection, and can improve the efficiency of production safety management. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0059] Figure 1 It is a flow chart of an industrial production safety data management method based on visual inspection of the present invention;

[0060] Figure 2 It is a structural schematic diagram of an industrial production safety data management system based on visual inspection of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] See also Figure 1 The present invention provides an industrial production safety data management method based on visual inspection, comprising the following steps:

[0063] S100, collect the task plans of all automation equipment in the production area, collect images of the automation equipment at different angles through cameras, build a three-dimensional scene and generate a device model for each automation equipment.

[0064] S200, dividing the working space area according to the equipment model in the three-dimensional scene, generating an object model through the personnel image, analyzing the positional relationship between the object model and the working space area to calculate the first coefficient and identify the risk object model.

[0065] S300, marking the automation equipment that has a spatial overlap area with the risk object model, establishing a predicted spatial area for the marked automation equipment and calculating a second coefficient, obtaining an adjustment device based on the second coefficient and adjusting its task plan.

[0066] S400, display the object model and work space area in the three-dimensional scene through a visual interface, and automatically warn the staff when a safety hazard occurs.

[0067] In S100, the production area refers to the work area used for industrial production. Automation equipment refers to mechanical equipment that completes specific process tasks in an automated manner. Task plan refers to the task that the automation equipment is performing, including the code and execution time of each action. Action refers to the basic motion unit of the automation equipment. The execution time refers to the time required for the automation equipment to perform the action. The automation equipment executes each action in sequence to complete the task plan. A three-dimensional scene refers to a three-dimensional space used to simulate the volume size and relative position of all automation equipment in the production area. The equipment model refers to a three-dimensional diagram used to simulate the automation equipment. Through the cameras installed at various angles in the production area, image acquisition is performed simultaneously, and common feature points in adjacent images are extracted for matching and marking of repeated areas and the same automation equipment. A multi-view three-dimensional reconstruction algorithm is used to combine images from multiple perspectives to build a three-dimensional scene, and a device model is generated for each automation equipment in the three-dimensional scene.

[0068] In S200, the specific steps are as follows:

[0069] S201, establish an action set for each automation device, and DM the code of the action being executed in the task plan now Put it into the corresponding action set. Set the prediction duration U z , when the sum of the execution time of all the codes corresponding to the actions in the action set is less than U z When the DM now The next code is put into the corresponding action set until the sum of the execution time of all the code corresponding actions in each action set is greater than or equal to U z .

[0070] S202, analyze the spatial position change of the automation equipment caused by each action in the action set, simulate the three-dimensional space occupied by the equipment model when executing the action in the corresponding action set in the three-dimensional scene, and combine the three-dimensional spaces of all actions in the action set as the working space area of ​​the corresponding automation equipment. Use the YOLOv5 human body detection algorithm to analyze the image information captured by each camera in real time. When the presence of human objects is detected in the production area, an object model is generated at the corresponding position in the three-dimensional scene through multi-view images, and the dynamic information of the human objects in reality is mapped to the object model in real time.

[0071] Each action set can only generate one workspace area. The three-dimensional spaces occupied by all actions in the action set are combined to form the workspace area. If the action is static during the duration, the corresponding three-dimensional space is the three-dimensional area occupied by the device model when it maintains the action. If the action is in motion during the duration, the corresponding three-dimensional space is the three-dimensional area that the device model passes through when executing the action.

[0072] The generation of object models is a virtual mapping of the real environment. The object model is generated in the three-dimensional scene according to the size and position of each person in the production area. The behavior and position changes of the personnel are obtained in real time and synchronously mapped to the three-dimensional scene for data analysis.

[0073] S203, identify the spatial overlap area between the object model and the work space area in the three-dimensional scene, and mark the object model with the spatial overlap area. Use the key point detection algorithm to identify different parts of the marked object model and mark the key points of the head. Combined with the head posture detection algorithm, analyze the spatial position change of the head key points to obtain the sight direction S of the marked object model. The spatial position coordinates (X h ,Y h ,Z h ) as the starting point and S as the direction to establish a vector

[0074] The spatial overlapping area represents the expected collision area between personnel and equipment. The object model is a real-time mapped model. The working space area includes the currently occupied three-dimensional area and the three-dimensional area expected to be occupied in the future.

[0075] S204, setting angle threshold E, real-time analysis of vector The spatial position coordinates (X k ,Y k ,Z k ) between them. When the angle is less than E, record the duration time when the angle remains unchanged and the time (X h ,Y h ,Z h ) and (X k ,Y k ,Z k ). Substitute the formula to calculate the first coefficient of each marked object model. Set the first coefficient threshold C, and take the marked object model with a first coefficient less than C as the risk object model. The first coefficient calculation formula is as follows:

[0076]

[0077] In the formula, XS first is the first coefficient, is the duration of the g-th working space area at the i-th time. is the g-th operation space area in the i-th time (X h ,Y h ,Z h ) and (X k ,Y k ,Z k ) is the average spatial distance between them. is the angle of the g-th working space area at the i-th time.

[0078] The first coefficient indicates the degree of attention paid by personnel to the automated equipment. When the automated equipment is likely to collide with the personnel's current location, the smaller the angle between the personnel's line of sight and the center point of the collision zone, the longer the duration, and the shorter the distance, the more it indicates that the personnel are more concerned about the collision that has not occurred. The lower the possibility of an accident, the smaller the first coefficient.

[0079] In S300, the specific steps are as follows:

[0080] S301: Obtaining the spatial overlap volume VO of the risk object model and each operation space area sum , calculate the average volume VO of all spatial overlapping areas corresponding to each risk object model ave . Mark the automation equipment corresponding to the work space area where there is spatial overlap. Set the prediction duration set {U1,U2,...,U z}, generate an action set for each element in the predicted duration set according to the steps in S201, and combine the three-dimensional spaces of all actions in each action set as the corresponding marked automation equipment in {U1, U2, ..., U z} when the predicted space area.

[0081] The number of prediction space regions is z, each prediction duration corresponds to an action set, each action set corresponds to a prediction space region, and the volume of the prediction space region is proportional to the prediction duration.

[0082] S302: Re-identify the spatial overlap area volume VC between the risk object model in the three-dimensional scene and each predicted spatial area under the corresponding marking automation device r , calculate the average volume VC of these spatial overlapping areas ave . Analyze the spatial position coordinates of the head center of the risk object model (X n ,Y n ,Z n ) and the spatial distance DI between the spatial position coordinates of the center point of the spatial overlapping area r, calculate the average value DI of these spatial distances ave Substitute the formula to calculate the second coefficient of each marking automation device, set the second coefficient threshold B, and use the marking automation device with a second coefficient greater than B as the adjustment device. The second coefficient calculation formula is as follows:

[0083]

[0084] In the formula, XS second is the second coefficient and α is a constant.

[0085] The second coefficient indicates the degree of danger of the automated equipment. When the volume of the spatial overlap area of ​​the automated equipment predicted spatial area increases over time, and the spatial distance from the risk object model increases over time, the higher the second coefficient is, the more automated adjustments are needed to avoid collisions between automated equipment and personnel.

[0086] S303, feeding back the adjustment device information to the operation center and controlling the adjustment device to adjust the task plan, so that the work space area generated by the new task plan does not have an intersecting spatial overlapping area with any object model in the three-dimensional scene.

[0087] The adjustment task plan is autonomously executed by the program pre-set for the automation equipment. Each time the automation equipment adjusts the task plan, it automatically feeds back to the operation center. The operation center can intervene or not take any action based on the feedback information.

[0088] In S400, the positional relationship between each object model and the working space area in the three-dimensional scene is displayed in the form of dynamic images through the visual interface of the operation center, highlighting the adjustment equipment with safety hazards and the new task plan after automatic adjustment, and warning the staff to determine whether to intervene.

[0089] See also Figure 2 The present invention provides an industrial production safety data management system based on visual detection, including a data acquisition module, a visual perception module, a safety management module and a visual early warning module.

[0090] The data acquisition module is used to collect images of the production area and the task plan of the automation equipment. The visual perception module is used to build a three-dimensional scene, generate equipment models based on the automation equipment and divide the work space area, generate object models through personnel images, calculate the first coefficient of the object model and identify the risk object model. The safety management module marks the automation equipment with overlapping spatial areas, establishes a predicted spatial area for the marked automation equipment and calculates the second coefficient, obtains the adjustment equipment according to the second coefficient and adjusts its task plan. The visual warning module is used to display the object model and work space area in the three-dimensional scene, and automatically warns the staff when a safety hazard occurs.

[0091] The data acquisition module includes an image information acquisition unit and a task plan acquisition unit.

[0092] The image information acquisition unit acquires images through cameras installed at different angles in the production area. The production area refers to the operating area used for industrial production.

[0093] The task plan collection unit is used to collect the task plan of each automation equipment. Automation equipment refers to mechanical equipment that completes specific process tasks in an automated manner. The task plan refers to the task that the automation equipment is performing, including the code and execution time of each action. The action refers to the basic motion unit of the automation equipment, and the execution time refers to the time required for the automation equipment to perform the action. The automation equipment executes each action in sequence to complete the task plan.

[0094] The visual perception module includes a region division unit and a behavior analysis unit.

[0095] The area division unit is used to build a three-dimensional scene and divide it into areas.

[0096] First, collect images from various angles of the production area, extract common feature points from adjacent images for matching, and mark repeated areas and the same automation equipment. Combine images from multiple perspectives to build a three-dimensional scene, and generate a device model for each automation equipment in the three-dimensional scene.

[0097] Secondly, establish an action set for each automated device and set the prediction duration U z , get the code DM of the action being executed in the task plan now , in order of execution from DM now Start putting the following codes into the corresponding action sets one by one, until the sum of the execution time of all the codes corresponding to the actions in each action set is greater than or equal to U z .

[0098] Finally, the 3D space occupied by the equipment model when executing the corresponding action set is simulated in the 3D scene, and the 3D space of all actions in the action set is combined as the working space area of ​​the corresponding automation equipment. The images taken by each camera are analyzed in real time. When the presence of personnel objects in the production area is detected, an object model is generated at the corresponding position in the 3D scene, and the dynamic information of the personnel objects in reality is mapped to the object model in real time.

[0099] The behavior analysis unit is used to calculate a first coefficient of the object model and identify a risky object model.

[0100] First, the spatial overlap area between the object model and the work space in the three-dimensional scene is identified, and the object model with the spatial overlap area is marked. The key point detection algorithm is combined with the head posture detection algorithm to analyze the spatial position change of the key points of the marked object model head to obtain the line of sight direction S. The spatial position coordinates of the center of the marked object model head (X h ,Y h ,Z h ) as the starting point and S as the direction to establish a vector

[0101] Secondly, set the angle threshold E and analyze the vector in real time. The spatial position coordinates (X k ,Y k ,Z k ). When the angle is less than E, record the duration of each time when the angle remains unchanged and the time (X h ,Y h ,Z h ) and (X k ,Y k ,Z k ) is the average spatial distance between them.

[0102] Finally, through the formula Calculate the first coefficient of each marked object model. Set the first coefficient threshold C, and take the marked object model with a first coefficient less than C as the risk object model. first is the first coefficient, is the duration of the g-th operating space area at the i-th time; is the g-th operation space area in the i-th time (X h ,Y h ,Z h ) and (X k ,Y k ,Z k ) between the average spatial distance; is the angle of the g-th working space area at the i-th time.

[0103] The safety management module includes a safety prediction unit and an emergency management unit.

[0104] The safety prediction unit is used to mark the automation equipment to establish a prediction space area and calculate the second coefficient.

[0105] First, obtain the spatial overlap volume VO of the risk object model and each operation space area sum , calculate the average volume VO of all spatial overlapping areas corresponding to each risk object model ave; Mark the automation equipment corresponding to the working space area where there is spatial overlap.

[0106] Secondly, set the prediction duration set {U1,U2,...,U z}, generate an action set for each element in the prediction duration set, and combine the three-dimensional spaces of all actions in each action set as the corresponding labeled automation equipment in {U1,U2,...,U z} when the predicted space area.

[0107] Then, the spatial overlap area volume VC between the risk object model in the three-dimensional scene and each predicted spatial area under the corresponding marking automation equipment is identified again r , calculate the average volume VC of these spatial overlapping areas ave ; Analyze the spatial position coordinates of the head center of the risk object model (X n ,Y n ,Z n ) and the spatial distance DI between the spatial position coordinates of the center point of the spatial overlapping area r , calculate the average value DI of these spatial distances ave .

[0108] Finally, substitute into the formula , calculate the second coefficient of each marking automation device, set the second coefficient threshold B, and use the marking automation device with a second coefficient greater than B as the adjustment device; wherein XS second is the second coefficient and α is a constant.

[0109] The emergency management unit will adjust the equipment information and feed it back to the operation center and control the equipment to adjust the task plan so that the work space area generated by the new task plan does not have any intersecting spatial overlapping area with any object model in the three-dimensional scene.

[0110] The visual warning module displays the positional relationship between each object model and the work space area in the three-dimensional scene through the visual interface of the operation center, displays the adjusted equipment and the adjusted new task plan, and warns the staff.

[0111] Embodiment 1:

[0112] Assume that there is a spatial overlap between the marked object model and the two working space areas A1 and A2. When the angle threshold is 90°, the following five observation records are collected for the marked object model:

[0113] Observation record 1: A1 spatial overlap area; angle 30°; duration 2s; average spatial distance 0.5m;

[0114] Observation record 2: A1 spatial overlap area; angle 120°; duration 5s; average spatial distance 0.3m;

[0115] Observation record 3: A2 spatial overlap area; angle 60°; duration 3s; average spatial distance 0.2m;

[0116] Observation record 4: A2 spatial overlap area; angle 120°; duration 2s; average spatial distance 0.3m;

[0117] Observation record 5: A2 spatial overlap area; angle 30°; duration 1s; average spatial distance 0.2m;

[0118] Substitute the formula to calculate the first coefficient of the labeled object model:

[0119]

[0120] Then the first coefficient of the labeled object model is 0.55.

[0121] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0122] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for industrial production safety data management based on visual inspection, characterized in that: The method comprises the following steps: S100, collect the task plans of all automation equipment in the production area, collect images of automation equipment from different angles through cameras, build a three-dimensional scene and generate a device model for each automation equipment; S200, dividing the working space area according to the equipment model in the three-dimensional scene, generating an object model through the personnel image, analyzing the positional relationship between the object model and the working space area, calculating a first coefficient, and identifying a risk object model; S300, marking the automation equipment in the spatial overlap area with the risk object model, establishing a predicted spatial area for the marked automation equipment and calculating a second coefficient, obtaining an adjustment device according to the second coefficient and adjusting its task plan; S400, display the object model and work space area in the three-dimensional scene through a visual interface, and automatically warn the staff when a safety hazard occurs.

2. The method for industrial production safety data management based on visual inspection according to claim 1, characterized in that: In S100, the production area refers to the work area used for industrial production; the automation equipment refers to the mechanical equipment that completes specific process tasks in an automated manner; the task plan refers to the task being performed by the automation equipment, including the code and execution time of each action, the action refers to the basic motion unit of the automation equipment, the execution time refers to the time required for the automation equipment to perform the action, and the automation equipment executes each action in sequence to complete the task plan; the three-dimensional scene refers to the three-dimensional space used to simulate the volume size and relative position of all automation equipment in the production area, and the equipment model refers to the three-dimensional diagram used to simulate the automation equipment; through the cameras installed at various angles in the production area to simultaneously collect images, extract the common feature points in adjacent images for matching and mark the repeated areas and the same automation equipment, use the multi-view three-dimensional reconstruction algorithm to combine the images of multiple viewpoints to build a three-dimensional scene, and generate a device model for each automation equipment in the three-dimensional scene.

3. The method for industrial production safety data management based on visual inspection according to claim 2, characterized in that: In S200, the specific steps are as follows: S201, establish an action set for each automation device, and DM the code of the action being executed in the task plan now Put it into the corresponding action set; set the prediction duration U z , when the sum of the execution time of all the codes corresponding to the actions in the action set is less than U z When the DM now The next code is put into the corresponding action set until the sum of the execution time of all the code corresponding actions in each action set is greater than or equal to U z ; S202, analyzing the spatial position change of the automation equipment caused by each action in the action set, simulating the three-dimensional space occupied by the equipment model when executing the action in the corresponding action set in the three-dimensional scene, and combining the three-dimensional spaces of all actions in the action set as the working space area of ​​the corresponding automation equipment; using the YOLOv5 human body detection algorithm to analyze the image information captured by each camera in real time, and when it is detected that there is a human object in the production area, an object model is generated at the corresponding position in the three-dimensional scene through multi-view images, and the dynamic information of the human object in reality is mapped to the object model in real time; S203, identifying the spatial overlap area between the object model and the working space area in the three-dimensional scene, marking the object model with the spatial overlap area; using the key point detection algorithm to identify different parts of the marked object model and mark the key points of the head, combining the head posture detection algorithm to analyze the spatial position change of the head key points to obtain the sight line direction S of the marked object model; using the spatial position coordinates (X h ,Y h ,Z h ) as the starting point and S as the direction to establish a vector S204, setting angle threshold E, real-time analysis of vector The spatial position coordinates (X k ,Y k ,Z k ) between them. When the angle is less than E, record the duration time when the angle remains unchanged and the time (X h ,Y h ,Z h ) and (X k ,Y k ,Z k ) between them; substituting the formula into the formula to calculate the first coefficient of each marked object model; setting the first coefficient threshold C, and taking the marked object model with a first coefficient less than C as the risk object model; The first coefficient is calculated as follows: In the formula, XS first is the first coefficient, is the duration of the g-th operating space area at the i-th time; is the g-th operation space area at the i-th time (X h ,Y h ,Z h ) and (X k ,Y k ,Z k ) between the average spatial distance; is the angle of the g-th working space area at the i-th time.

4. The method for industrial production safety data management based on visual inspection according to claim 3 is characterized in that: In S300, the specific steps are as follows: S301: Obtaining the spatial overlap volume VO of the risk object model and each operation space area sum , calculate the average volume VO of all spatial overlapping areas corresponding to each risk object model ave ; Mark the automation equipment corresponding to the work space area where there is spatial overlap; Set the prediction duration set {U1,U2,...,U z }, generate an action set for each element in the predicted duration set according to the steps in S201, and combine the three-dimensional spaces of all actions in each action set as the corresponding marked automation equipment in {U1, U2, ..., U z }Prediction space area; S302: Re-identify the spatial overlap area volume VC between the risk object model in the three-dimensional scene and each predicted spatial area under the corresponding marking automation device r , calculate the average volume VC of these spatial overlapping areas ave ; Analyze the spatial position coordinates of the head center of the risk object model (X n ,Y n ,Z n ) and the spatial distance DI between the spatial position coordinates of the center point of the spatial overlapping area r , calculate the average value DI of these spatial distances ave ; Substitute the formula to calculate the second coefficient of each marking automation device, set the second coefficient threshold B, and use the marking automation device with a second coefficient greater than B as the adjustment device; The second coefficient calculation formula is as follows: In the formula, XS second is the second coefficient, α is a constant; S303, feeding back the adjustment device information to the operation center and controlling the adjustment device to adjust the task plan, so that the work space area generated by the new task plan does not have an intersecting spatial overlapping area with any object model in the three-dimensional scene.

5. The method for industrial production safety data management based on visual inspection according to claim 4 is characterized in that: In S400, the positional relationship between each object model and the working space area in the three-dimensional scene is displayed in the form of dynamic images through the visual interface of the operation center, highlighting the adjustment equipment with safety hazards and the new task plan after automatic adjustment, and warning the staff to determine whether to intervene.

6. An industrial production safety data management system based on visual inspection, characterized by: The system includes a data acquisition module, a visual perception module, a safety management module and a visual warning module; The data acquisition module is used to collect images of the production area and task plans of the automation equipment; The visual perception module is used to build a three-dimensional scene, generate an equipment model according to the automated equipment and divide the working space area, generate an object model through the personnel image, calculate the first coefficient of the object model and identify the risk object model; the safety management module marks the automated equipment with overlapping spatial areas, establishes a predicted spatial area for the marked automated equipment and calculates the second coefficient, obtains the adjustment equipment according to the second coefficient and adjusts its task plan; The visual warning module is used to display object models and work space areas in three-dimensional scenes, and automatically warn staff when safety hazards occur.

7. The industrial production safety data management system based on visual inspection according to claim 6 is characterized in that: The data acquisition module includes an image information acquisition unit and a task plan acquisition unit; The image information acquisition unit acquires images through cameras installed at different angles in the production area. The production area refers to the operation area used for industrial production. The task plan collection unit is used to collect the task plan of each automation equipment; automation equipment refers to mechanical equipment that completes specific process tasks in an automated manner; task plan refers to the task being performed by the automation equipment, including the code and execution time of each action, action refers to the basic motion unit of the automation equipment, execution time refers to the time required for the automation equipment to perform the action, and the automation equipment executes each action in sequence to complete the task plan.

8. The industrial production safety data management system based on visual inspection according to claim 7 is characterized in that: The visual perception module includes a region division unit and a behavior analysis unit; The area division unit is used to build a three-dimensional scene and divide the area; First, collect images from various angles of the production area, extract common feature points from adjacent images for matching, and mark repeated areas and identical automation equipment. Combine images from multiple perspectives to build a 3D scene, and generate a device model for each automation equipment in the 3D scene. Secondly, establish an action set for each automated device and set the prediction duration U z , get the code DM of the action being executed in the task plan now , in order of execution from DM now Start putting the following codes into the corresponding action sets one by one, until the sum of the execution time of all the codes corresponding to the actions in each action set is greater than or equal to U z ; Finally, the three-dimensional space occupied by the equipment model when executing the actions in the corresponding action set is simulated in the three-dimensional scene, and the three-dimensional spaces of all actions in the action set are combined as the working space area of ​​the corresponding automation equipment; Analyze the images taken by each camera in real time. When a person is detected in the production area, an object model is generated at the corresponding position in the 3D scene. The dynamic information of the person in reality is mapped to the object model in real time. The behavior analysis unit is used to calculate the first coefficient of the object model and identify the risk object model; Firstly, the spatial overlap area between the object model and the working space in the 3D scene is identified, and the object model with the spatial overlap area is marked. The key point detection algorithm combined with the head posture detection algorithm is used to analyze the spatial position change of the key points of the marked object model's head to obtain the line of sight direction S. The spatial position coordinates (X h ,Y h ,Z h ) as the starting point and S as the direction to establish a vector Secondly, set the angle threshold E and analyze the vector in real time. The spatial position coordinates (X k ,Y k ,Z k ) between them; when the angle is less than E, record the duration time when the angle remains unchanged and the time (X h ,Y h ,Z h ) and (X k ,Y k ,Z k ) between the average spatial distance; Finally, through the formula Calculate the first coefficient of each marked object model; set the first coefficient threshold C, and take the marked object model with a first coefficient less than C as the risk object model; wherein XS first is the first coefficient, is the duration of the g-th operating space area at the i-th time; is the g-th operation space area at the i-th time (X h ,Y h ,Z h ) and (X k ,Y k ,Z k ) between the average spatial distance; is the angle of the g-th working space area at the i-th time.

9. The industrial production safety data management system based on visual inspection according to claim 8, characterized in that: The safety management module includes a safety prediction unit and an emergency management unit; The safety prediction unit is used to mark the automation equipment to establish a prediction space area and calculate the second coefficient; First, obtain the spatial overlap volume VO of the risk object model and each operation space area sum , calculate the average volume VO of all spatial overlapping areas corresponding to each risk object model ave ; Mark the automation equipment corresponding to the work space area where there is spatial overlap; Secondly, set the prediction duration set {U1,U2,...,U z }, generate an action set for each element in the prediction duration set, and combine the three-dimensional spaces of all actions in each action set as the corresponding labeled automation equipment in {U1,U2,...,U z }Prediction space area; Then, the spatial overlap area volume VC between the risk object model in the three-dimensional scene and each predicted spatial area under the corresponding marking automation equipment is identified again r , calculate the average volume VC of these spatial overlapping areas ave ; Analyze the spatial position coordinates of the head center of the risk object model (X n ,Y n ,Z n ) and the spatial distance DI between the spatial position coordinates of the center point of the spatial overlapping area r , calculate the average value DI of these spatial distances ave ; Finally, substitute into the formula , calculate the second coefficient of each marking automation device, set the second coefficient threshold B, and use the marking automation device with a second coefficient greater than B as the adjustment device; wherein XS second is the second coefficient, α is a constant; The emergency management unit will adjust the equipment information to the operation center and control the equipment to adjust the task plan so that the work space area generated by the new task plan does not have any intersecting spatial overlapping area with any object model in the three-dimensional scene.

10. An industrial production safety data management system based on visual inspection according to claim 9, characterized in that: The visual warning module displays the positional relationship between each object model and the work space area in the three-dimensional scene through the visual interface of the operation center, displays the adjusted equipment and the adjusted new task plan, and warns the staff.

Images