Safety protection methods, devices, equipment, and storage media for workbenches based on planar vision

Abstract

This invention relates to the field of automation technology, and more particularly to a method, apparatus, equipment, and storage medium for workbench safety protection based on planar vision. This application captures images of the work area using a multi-view monitoring camera, identifies and tracks moving objects within the work area using a deep learning model, determines potential collision targets, and then predicts the movement trajectory of the target within a few seconds based on the image bounding box of the target over a certain time period. This trajectory is compared with the projected movement range of the robotic arm within those seconds to predict potential collision risks. Control commands are then generated to intelligently control the start and stop states of the robotic arm, ensuring safety in the human-machine collaborative environment. The implementation of this solution significantly improves the safety of human-machine collaboration in industrial environments. Through real-time monitoring and intelligent prediction of collision risks, it effectively reduces personnel injuries and equipment damage caused by accidental collisions within the work area, providing an innovative safety protection solution for the field of intelligent manufacturing.

Description

Technical Field

[0001] This invention relates to the field of automation technology, and in particular to a method, apparatus, equipment and storage medium for workbench safety protection based on planar vision. Background Technology

[0002] With the global adoption of the concept of smart manufacturing, the manufacturing industry is undergoing a profound technological transformation. Smart manufacturing systems can monitor production processes in real time, automatically adjust production plans, optimize resource allocation, and improve production efficiency and product quality. Against this backdrop, factories and warehouses are increasingly deploying automated equipment and robots to reduce labor costs, increase production efficiency and flexibility, while meeting personalized and customized production needs.

[0003] In the advancement of intelligent manufacturing, human-robot collaboration has become an important development direction. Human-robot collaboration allows workers and robots to work together in the same work area, leveraging their respective strengths to improve production efficiency and flexibility. However, this collaborative model also brings new safety challenges. Traditional industrial robots are usually isolated in specific work areas to prevent direct contact with workers. But in a human-robot collaborative environment, robots need to operate near workers, which increases the risk of collisions between robots and workers.

[0004] Therefore, how to effectively predict and prevent collisions between robots and workers has become an urgent problem to be solved in this field. Summary of the Invention

[0005] The main objective of this invention is to provide a workbench safety protection method, device, equipment, and storage medium based on planar vision, aiming to solve the technical problem of how to effectively predict and prevent collisions between robots and workers in the prior art.

[0006] To achieve the above objectives, the present invention provides a workbench safety protection method based on planar vision, the method comprising the following steps:

[0007] Based on multi-view images of the working area, determine the collision detection targets within the working area;

[0008] Obtain the image bounding box of the collision detection target in the working area image from various viewpoints;

[0009] The collision prediction result is obtained by comparing the projection of the robotic arm's range of motion with the image bounding box of the collision detection target.

[0010] Based on the collision prediction results, control commands for the robotic arm are generated to control the start and stop states of the robotic arm.

[0011] Optionally, determining the collision detection target within the working area based on multi-view working area images includes:

[0012] The working area images are acquired from multiple perspectives, and the spatial positions of the multiple monitoring cameras satisfy a surround layout.

[0013] Based on the moving object detection model, the moving object recognition results are obtained by identifying multiple consecutive frames of working area images from the same viewpoint.

[0014] The results of moving object recognition from various perspectives are summarized and screened to determine the collision detection targets within the working area.

[0015] Optionally, the step of summarizing and screening the moving object recognition results from various perspectives to determine the collision detection targets within the working area includes:

[0016] Based on the moving object recognition results from the perspective of each surveillance camera and the spatial position information of the corresponding surveillance camera, the spatial position of each moving object is obtained.

[0017] If the spatial position of the moving object coincides with the range of the material area, then the object is determined to be an object in the material area.

[0018] If the spatial position of the moving object coincides with the working area of ​​the robotic arm, then the object is determined to be a robotic arm.

[0019] By eliminating material objects and robotic arms from the moving object, collision detection targets within the working area are obtained.

[0020] Optionally, obtaining the collision prediction result based on the image bounding box of the collision detection target projected onto the robotic arm's range of motion includes:

[0021] The image bounding boxes of the collision-detected objects within a preset time period are summarized to obtain a set of image bounding box positions.

[0022] Based on the set of image bounding box positions, the predicted trajectory of the collision-detected object under the viewpoints of each monitoring camera is obtained;

[0023] Based on the predicted trajectory of the collision-detected object, the projection of the collision-detected object is obtained;

[0024] The collision prediction result is obtained based on the projection of the collision-detected object and the projection of the robotic arm's range of motion.

[0025] Optionally, obtaining the collision prediction result based on the projection of the collision-detected object and the projection of the robotic arm's range of motion includes:

[0026] The collision detection object projection from the same surveillance camera viewpoint is compared frame by frame with the robotic arm motion range projection to obtain the time point when the two projections intersect.

[0027] If the number of surveillance camera viewpoints that generate projection intersection at the same time point is greater than or equal to the preset number, then it is determined that there is a collision risk within the predicted trajectory duration.

[0028] If the number of surveillance camera viewpoints that generate projection intersection at the same time point is less than the preset number, it is determined that there is no risk of collision within the predicted trajectory duration.

[0029] Optionally, before obtaining the collision prediction result based on the image bounding box of the collision detection target projected from the robot arm's range of motion, the method further includes:

[0030] Obtain the robotic arm's task motion instructions;

[0031] The spatial motion trajectory of the robotic arm is obtained according to the task action command of the robotic arm;

[0032] The spatial motion trajectory is decomposed frame by frame to obtain the spatial position of the robotic arm at each moment.

[0033] Based on the spatial position of the robotic arm, obtain the bounding box of the robotic arm from the corresponding viewpoint of each monitoring camera;

[0034] The bounding boxes of the robotic arm from the same surveillance camera perspective are summarized to obtain the projection of the robotic arm's motion range.

[0035] Optionally, generating robotic arm control commands based on the collision prediction results to control the start and stop states of the robotic arm includes:

[0036] If the collision prediction result indicates a collision risk, a pause command is sent to the robotic arm, and a warning light is activated until the collision detection target leaves the robotic arm's range of motion.

[0037] If the collision prediction result indicates that there is no collision risk, the robot arm continues to perform its scheduled task while continuously monitoring the work area to update the collision prediction result in real time.

[0038] Furthermore, to achieve the above objectives, the present invention also proposes a workbench safety protection device based on planar vision, the workbench safety protection device based on planar vision comprising:

[0039] The target recognition module is used to determine the collision detection targets within the working area based on multi-view images of the working area;

[0040] The image processing module is used to acquire the image bounding box of the collision detection target in the working area image from various viewpoints;

[0041] The collision prediction module is used to obtain the collision prediction result based on the image bounding box of the robot arm's motion range projection and the collision detection target;

[0042] The control module is used to generate control commands for the robotic arm based on the collision prediction results, so as to control the start and stop states of the robotic arm.

[0043] Furthermore, to achieve the above objectives, the present invention also proposes a workbench safety protection device based on planar vision. The workbench safety protection device based on planar vision includes: a memory, a processor, and a workbench safety protection program based on planar vision stored in the memory and executable on the processor. The workbench safety protection program based on planar vision is configured to implement the steps of the workbench safety protection method based on planar vision as described above.

[0044] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a workbench safety protection program based on planar vision, wherein when the workbench safety protection program based on planar vision is executed by a processor, it implements the steps of the workbench safety protection method based on planar vision as described above.

[0045] The present application proposes one or more technical solutions that have at least the following technical effects: This application captures images of the work area using a multi-view monitoring camera, utilizes a deep learning model to identify and track moving objects within the work area, determines potential collision targets, and then predicts the motion trajectory within a few seconds based on the image bounding box of the detected target over a certain time period. This trajectory is compared with the projected motion range of the robotic arm within those seconds to predict potential collision risks. Based on this, control commands are generated to intelligently control the start and stop states of the robotic arm, ensuring the safety of the human-machine collaborative environment. The implementation of this solution significantly improves the safety of human-machine collaboration in industrial environments. Through real-time monitoring and intelligent prediction of collision risks, it effectively reduces personnel injuries and equipment damage caused by accidental collisions within the work area, providing an innovative safety protection solution for the field of intelligent manufacturing. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating the first embodiment of the workbench safety protection method based on planar vision according to the present invention.

[0049] Figure 2 This is a flowchart illustrating the second embodiment of the workbench safety protection method based on planar vision according to the present invention.

[0050] Figure 3 This is a schematic diagram of a human-machine collaborative workbench for the workbench safety protection method based on planar vision according to the present invention;

[0051] Figure 4 This is a flowchart illustrating the third embodiment of the workbench safety protection method based on planar vision of the present invention.

[0052] Figure 5 This is a structural block diagram of the first embodiment of the workbench safety protection device based on planar vision of the present invention;

[0053] Figure 6 This is a schematic diagram of the structure of a workbench safety protection device based on planar vision, which is part of the hardware operating environment involved in the embodiments of the present invention.

[0054] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0056] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0057] The main solution of this application embodiment is: to determine the collision detection target in the working area based on the multi-view working area image; to obtain the image bounding box of the collision detection target in the working area image under each view; to obtain the collision prediction result based on the projection of the robotic arm's motion range and the image bounding box of the collision detection target; and to generate robotic arm control commands based on the collision prediction result to control the start and stop state of the robotic arm.

[0058] Currently, some human-machine collaborative automation workbenches are equipped with robotic arms. On these workbenches, workers and mechanical equipment work together, so it is necessary to ensure that the robotic arms do not collide with the workers' hands or other products or parts during normal operation.

[0059] Traditional technologies typically involve setting up physical barriers or protective fences around the robotic arm to isolate its working area from the worker's area of ​​activity, or using safety light curtains to monitor the robotic arm's working area. When the light curtain is blocked, the robotic arm automatically stops moving. However, safety light curtains can only be set in fixed areas, making it very difficult to implement different protective measures for different tasks. While these traditional technologies improve the safety of human-robot collaborative environments to some extent, they usually have certain limitations. For example, physical isolation limits flexibility, sensors may have blind spots, and they mostly rely on passive protection rather than active prediction and avoidance of collisions.

[0060] This application provides a solution that captures images of the work area using multi-view monitoring cameras, utilizes a deep learning model to identify and track moving objects within the work area, determines potential collision targets, and then predicts the trajectory of the detected targets within a few seconds based on the image bounding boxes over a certain time period. This trajectory is then compared with the projected movement range of a robotic arm within those seconds to predict potential collision risks. Based on this, control commands are generated to intelligently control the start and stop states of the robotic arm, ensuring the safety of the human-robot collaborative environment. The implementation of this solution significantly improves the safety of human-robot collaboration in industrial environments. Through real-time monitoring and intelligent collision risk prediction, it effectively reduces personnel injuries and equipment damage caused by accidental collisions within the work area, providing an innovative safety protection solution for the field of intelligent manufacturing.

[0061] Based on this, embodiments of the present invention provide a workbench safety protection method based on planar vision, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of a workbench safety protection method based on planar vision according to the present invention.

[0062] In this embodiment, the workbench safety protection method based on planar vision includes the following steps:

[0063] Step S10: Determine the collision detection targets within the working area based on the multi-view working area images.

[0064] It should be noted that the purpose of this step is to use multiple surveillance cameras in different locations to collect image data of the same work area, and to identify and locate objects in the work area that may collide with the robotic arm from the captured image data.

[0065] Understandably, at any given moment, surveillance cameras from different perspectives may only capture some images of objects that could potentially collide with the robotic arm. To avoid missing or misjudging objects that might collide with the robotic arm, multiple surveillance cameras are arranged in a ring around the material handling table, ensuring that every corner of the table is within the monitoring range of at least one camera. There is also an overlapping area of ​​view between adjacent surveillance cameras, which helps improve the accuracy of object detection and reduces misjudgments through cross-verification of multiple perspectives.

[0066] It should be understood that determining the collision detection target involves more than just detecting the existence of the object; it also includes the dynamic tracking and identification of the object. The movement of any object on the worktable can be continuously tracked by multiple cameras, even if the object moves at a high speed or changes position rapidly on the worktable. In addition, due to the characteristics of the worktable, in addition to the operator's hand and the robotic arm, there may be some workpieces on the material table itself that will move as the engineering operation progresses. Therefore, in the process of identifying the collision detection target, it is also necessary to exclude objects in the material area and the robotic arm itself, and only focus on other objects that may collide with the robotic arm.

[0067] Step S20: Obtain the image bounding box of the collision detection target in the working area image from various viewpoints.

[0068] It should be noted that this invention uses the YOLOv8 network model as the main framework of the detection model, and performs identification and classification training on objects that may enter the device before the workbench is officially run. It can be understood that the YOLOv8 network model is an advanced deep learning technology specifically designed for real-time object detection tasks. It can quickly and accurately identify and classify objects in images and provide precise bounding boxes for objects.

[0069] It should be understood that, over time, the bounding boxes formed by collision detection targets from various perspectives will move. This movement may bring them closer to the working swing area of ​​the robotic arm or it may move them away from the worktable.

[0070] Step S30: Obtain the collision prediction result based on the image bounding box of the robot arm's motion range projection and the collision detection target.

[0071] It should be noted that based on the movement of the collision detection target over a period of time, the possible trajectory in a short period of time can be roughly estimated. Generally, statistical or machine learning methods, such as linear regression, Gaussian processes, or more complex deep learning models, can be used to predict the future position of the object.

[0072] Understandably, this application uses the Kalman filter algorithm to estimate the dynamic state of an object, including its position and velocity, and to predict its state at the next moment. In the specific implementation process, the monitoring camera acquires images at a rate of 30 or 60 frames per second. Based on multiple frames of images captured in the past 3 seconds, the system predicts the position of the image bounding box of the collision detection target within the next 2 seconds. Generally, the prediction time of 2 seconds here is based on considerations of practical applications. This time window is neither too long to make it difficult to accurately predict rapidly changing dynamic environments, nor too short to be sufficient to take effective preventive measures.

[0073] It should be understood that as time goes on, the multiple frames of images of the collision detection target within the past 3 seconds will change over time, and the corresponding predicted trajectory will also change. Only when the predicted trajectory overlaps with the projection of the robotic arm's range of motion will it be considered to have a collision risk.

[0074] In one embodiment, before obtaining the collision prediction result based on the projection of the robotic arm's motion range and the image bounding box of the collision detection target, the method further includes: acquiring robotic arm task action instructions; obtaining the spatial motion trajectory of the robotic arm based on the robotic arm task action instructions; decomposing the spatial motion trajectory frame by frame to obtain the spatial position of the robotic arm at each time moment; acquiring the robotic arm bounding box of each monitoring camera at its corresponding viewpoint based on the robotic arm spatial position; and summarizing the robotic arm bounding boxes at the same monitoring camera viewpoint to obtain the projection of the robotic arm's motion range.

[0075] It should be noted that mapping the motion trajectory or working area of ​​a robotic arm in three-dimensional space onto a two-dimensional plane (usually the viewing plane of a monitoring camera) is a process and result that allows us to visualize and analyze the motion of the robotic arm on a plane, which is particularly important for safety monitoring and collision prevention systems.

[0076] It is understandable that since the robotic arm operates according to fixed task action instructions, its trajectory is predictable. That is, at any given moment, the robotic arm's movement over a period of time can be obtained based on the task action instructions. Based on this, the range of motion of the robotic arm under the view of each monitoring camera can be obtained, and the minimum bounding box surrounding this range can be determined.

[0077] It should be understood that if the predicted trajectory of a collision detection target at a certain moment indicates that it will overlap with the bounding box in the projection of the robotic arm's range of motion within the predicted time, then a collision should be considered possible within that time period. On the other hand, to reduce false detections, other perspectives will be queried simultaneously to see if bounding box overlap occurs at the same time. Only when the number of overlaps reaches the confidence threshold will a collision risk be considered.

[0078] Step S40: Based on the collision prediction results, generate robotic arm control commands to control the start and stop states of the robotic arm.

[0079] It should be noted that if a collision event is predicted, the robotic arm's movements will be paused until the detected target leaves the robotic arm and reaches a safe position before resuming its previous work.

[0080] In one embodiment, generating robotic arm control commands based on the collision prediction results to control the start and stop states of the robotic arm includes: if the collision prediction results indicate a collision risk, sending a pause command to the robotic arm and activating a warning light until the collision detection target leaves the movement range of the robotic arm; if the collision prediction results indicate no collision risk, continuing to execute the predetermined task of the robotic arm while continuously monitoring the work area to update the collision prediction results in real time.

[0081] Understandably, pausing the robotic arm's movement and illuminating the indicator light is a preventative measure designed to provide workers with a clear warning, ensuring they have sufficient time and space to handle potential hazards. Based on this control logic, the robotic arm can stop its movements promptly, ensuring the safety of workers and equipment, while quickly resuming normal operations after the risk has been eliminated, minimizing production interruptions. This dynamic safety control strategy is key to achieving efficient human-machine collaboration.

[0082] This embodiment determines the collision detection target within the working area based on multi-view working area images; obtains the image bounding box of the collision detection target in the working area images from various viewpoints; obtains the collision prediction result based on the projection of the robotic arm's motion range onto the image bounding box of the collision detection target; and generates robotic arm control commands based on the collision prediction result to control the start and stop states of the robotic arm.

[0083] In summary, this embodiment captures images of the work area using multi-view monitoring cameras, utilizes a deep learning model to identify and track moving objects within the work area, determines potential collision targets, and then predicts the trajectory of the detected targets within a few seconds based on the image bounding boxes over a certain time period. This trajectory is then compared with the projected movement range of the robotic arm within those seconds to predict potential collision risks. Based on this, control commands are generated to intelligently control the start and stop states of the robotic arm, ensuring the safety of the human-machine collaborative environment. The implementation of this solution significantly improves the safety of human-machine collaboration in industrial environments. Through real-time monitoring and intelligent collision risk prediction, it effectively reduces personnel injuries and equipment damage caused by accidental collisions within the work area.

[0084] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S10 further includes:

[0085] Step S101: Acquire images of the working area from multiple perspectives, wherein the spatial positions of the multiple monitoring cameras satisfy a surround layout.

[0086] like Figure 3 As shown, Figure 3 This is a schematic diagram of the human-machine collaborative workbench of the present invention, which includes the following components: an overall frame 1 for maintaining the overall stability of the device; a multi-joint collaborative robot 2 mounted upside down on the frame 1 to ensure the stability of the overall frame during movement; a working area 3 set in the frame 1 to ensure that the area is within the travel range of the robot 2; other structures for auxiliary production can be installed as needed; and several safety monitoring cameras 4 installed on the frame 1 at different positions and angles.

[0087] Step S102: Based on the moving object detection model, identify multiple consecutive frames of working area images from the same viewpoint to obtain the moving object recognition result.

[0088] It should be noted that by analyzing a series of images from the same viewpoint using a moving object detection model, moving objects within the working area can be identified and tracked. This process typically involves separating dynamic objects from a static background and determining their motion characteristics, such as their position, size, and possible direction of motion in the image.

[0089] It is understandable that even if moving objects identified from different viewpoints appear different, they may actually be the same object. Therefore, a data association process is needed to match and associate objects identified from different viewpoints to gain a complete understanding of each individual object. For example, if one camera identifies three moving objects and another camera from a different viewpoint identifies four moving objects, then the position of the moving objects in the image and the position of the camera in the workbench area can determine whether there is overlap between these moving objects.

[0090] Step S103: Summarize and screen the moving object recognition results from various perspectives to determine the collision detection targets within the working area.

[0091] It should be noted that, under normal circumstances, the system will integrate the moving object recognition results from multiple perspectives to construct a comprehensive and dynamically updated view of the work area. This includes tracking and recognizing all potential moving objects, meaning that the recognition process here includes the operator's limbs, the robotic arm, and the workpieces on the material platform.

[0092] Understandably, due to the characteristics of the material platform, the vertical height of objects on the material platform is fixed. Therefore, by observing the position of the moving object in the figure from multiple perspectives, it can be roughly determined whether the object is a workpiece on the material platform. If it is, then subsequent collision prediction and position tracking are unnecessary.

[0093] In one embodiment, the step of summarizing and screening the moving object recognition results from various perspectives to determine the collision detection targets within the working area includes: obtaining the spatial position of each moving object based on the moving object recognition results from the perspectives of each monitoring camera and the spatial position information of the corresponding monitoring camera; if the spatial position of the moving object coincides with the material area range, then the object is determined to be a material area object; if the spatial position of the moving object coincides with the working area of ​​the robotic arm, then the object is determined to be the robotic arm; and filtering out material area objects and robotic arms from the moving objects to obtain the collision detection targets within the working area.

[0094] It should be noted that by integrating the moving object recognition results from multiple perspectives and combining them with the spatial position information of the monitoring camera, the exact position of the object in the working area can be determined. Then, the objects in the material area and the robotic arm are screened out to avoid false detection, because these objects are usually known, fixed or controlled. The targets that need to be collision detected can be determined through the above screening process.

[0095] This embodiment acquires images of the working area from multiple perspectives, and the spatial positions of the multiple monitoring cameras satisfy a surround layout; based on a moving object detection model, multiple consecutive frames of working area images from the same perspective are identified to obtain moving object recognition results; the moving object recognition results from each perspective are summarized and screened to determine the collision detection targets within the working area.

[0096] In summary, this embodiment significantly improves the accuracy and efficiency of collision detection through precise spatial location determination and intelligent screening mechanisms. It not only reduces false detections and unnecessary safety interventions but also ensures the safety and reliability of the robotic arm operation in dynamically changing work environments. Furthermore, the real-time monitoring and dynamic updating capabilities of this solution enable the safety protection system to respond quickly to emergencies and take timely measures, thereby reducing safety risks in the work area, protecting the safety of workers and equipment, and improving production efficiency.

[0097] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 Step S30 includes:

[0098] Step S301: Summarize the image bounding boxes of the collision detection object within a preset time period to obtain a set of image bounding box positions.

[0099] It should be noted that the preset time period here refers to a period of time from the current moment to a few seconds ago. The span of this time window is related to the accuracy requirements of trajectory prediction.

[0100] Understandably, a surveillance camera will capture multiple images within a few seconds, and the positional changes of the collision detection object in the images will form a continuous trajectory of positional changes.

[0101] Step S302: Based on the set of image bounding box positions, obtain the predicted trajectory of the collision detection object from the perspective of each monitoring camera.

[0102] It should be noted that this step involves integrating data collected by multiple surveillance cameras. Each camera may provide different perspectives and information, so this data needs to be combined to obtain a comprehensive view of the object's motion.

[0103] Understandably, because an object's position changes under different camera viewpoints, this information can be used to construct the object's 3D trajectory. This typically involves spatial coordinate transformation and data fusion techniques to ensure that data obtained from different viewpoints can be accurately mapped to the same 3D space.

[0104] It should be understood that the accuracy of trajectory prediction depends not only on the data quality of individual cameras but also on the effectiveness of data integration from multiple cameras. This means that even if the data from a single camera is highly accurate, improper integration of data from different cameras can lead to inaccurate trajectory predictions. Therefore, this step requires highly precise data processing and analysis techniques to ensure that data obtained from multiple perspectives can be correctly integrated and interpreted.

[0105] Step S303: Obtain the projection of the collision detection object based on the predicted trajectory of the collision detection object.

[0106] It should be noted that since the predicted trajectory is a series of position changes over time, the position of the object in each frame can be extracted and mapped onto a common reference plane to form the projection of the collision detection object from the perspective of the monitoring camera.

[0107] Understandably, since there are multiple surveillance cameras at the same time, it is necessary to summarize the projections of collision-detected objects from multiple surveillance perspectives at the same moment. This can improve the accuracy of the information source used for target detection and tracking of the subsequent collision-detected object projections.

[0108] Step S304: Obtain the collision prediction result based on the projection of the collision-detected object and the projection of the robotic arm's range of motion.

[0109] In one embodiment, obtaining the collision prediction result based on the collision detection object projection and the robotic arm motion range projection includes: comparing the collision detection object projection and the robotic arm motion range projection frame by frame from the same monitoring camera viewpoint to obtain the time node where the two projections intersect; if the number of monitoring camera viewpoints that intersect at the same time node is greater than or equal to a preset number, it is determined that there is a collision risk within the predicted trajectory duration; if the number of monitoring camera viewpoints that intersect at the same time node is less than the preset number, it is determined that there is no collision risk within the predicted trajectory duration.

[0110] It's important to note that comparing the object's projection and the robotic arm's projection at the same point in time to determine if a potential collision exists requires precise time synchronization and spatial alignment to ensure accuracy. By comparing the object's projection and the robotic arm's projection frame-by-frame from the same monitoring camera's viewpoint, their relative spatial positions can be determined. This method allows the system to assess the probability of a collision at each point in time within the predicted trajectory's timeframe and make decisions accordingly.

[0111] Understandably, the preset number is a threshold used to determine whether there is sufficient evidence to suggest a collision risk. If multiple perspectives show an intersection at the same point in time, the likelihood of a collision can be predicted with greater confidence. This method utilizes information from multiple perspectives to enhance the reliability of the prediction. For example, when there are four surveillance cameras in the system, at least three cameras need to detect an intersection between their projections before a collision risk is considered to exist.

[0112] This embodiment constructs the 3D motion trajectory of an object by aggregating the bounding boxes of collision detection object images captured by multiple surveillance cameras within a preset time period, and projects this trajectory onto a 2D plane. By comparing the object projection frame by frame with the robotic arm's motion range projection, the system can predict potential collision risks. Specifically, the system compares the object projection from the same surveillance camera's viewpoint with the robotic arm projection, determines the time point where the intersection occurs, and judges whether a collision risk exists based on the number of surveillance camera views that intersect. This method improves the accuracy and reliability of collision detection because it integrates information from multiple perspectives, reducing the errors and uncertainties that may arise from a single perspective.

[0113] In summary, this embodiment achieves comprehensive monitoring and accurate prediction of object motion trajectories by integrating data from multiple monitoring cameras, thereby significantly improving the accuracy of collision detection. This method not only reduces false alarms and missed alarms but also provides real-time response in dynamic environments, ensuring the safety of robotic arm operations. By setting thresholds to determine collision risk, the system only issues warnings when potential collisions are consistently displayed from multiple perspectives, enhancing the reliability of the prediction results. Furthermore, this solution can adapt to different monitoring camera layouts and environmental conditions, exhibiting good flexibility and adaptability, and providing strong support for the safe operation of automated and robotic systems.

[0114] This application also provides a workbench safety protection device based on planar vision; please refer to [reference needed]. Figure 5 The workbench safety protection device based on planar vision includes:

[0115] The target recognition module 10 is used to determine the collision detection target within the working area based on multi-view working area images;

[0116] Image processing module 20 is used to acquire the image bounding box of the collision detection target in the working area image from various viewpoints;

[0117] The collision prediction module 30 is used to obtain the collision prediction result based on the image bounding box of the robot arm's motion range projection and the collision detection target;

[0118] The control module 40 is used to generate robotic arm control commands based on the collision prediction results, so as to control the start and stop states of the robotic arm.

[0119] In one embodiment, the target recognition module 10 is further configured to acquire working area images from multiple perspectives, wherein the spatial positions of the multiple monitoring cameras satisfy a surround layout; based on a moving object detection model, identify multiple consecutive frames of working area images from the same perspective to obtain moving object recognition results; and summarize and screen the moving object recognition results from each perspective to determine the collision detection targets within the working area.

[0120] In one embodiment, the target recognition module 10 is further configured to obtain the spatial position of each moving object based on the moving object recognition results from the perspective of each monitoring camera and the spatial position information of the corresponding monitoring camera; if the spatial position of the moving object coincides with the material area range, then the object is determined to be a material area object; if the spatial position of the moving object coincides with the working area of ​​the robotic arm, then the object is determined to be a robotic arm; and material area objects and robotic arms are filtered out from the moving objects to obtain the collision detection targets within the working area.

[0121] In one embodiment, the collision prediction module 30 is further configured to summarize the image bounding boxes of the collision-detected object within a preset time period to obtain an image bounding box position set; obtain the predicted trajectory of the collision-detected object from the perspective of each monitoring camera based on the image bounding box position set; obtain the projection of the collision-detected object based on the predicted trajectory of the collision-detected object; and obtain the collision prediction result based on the projection of the collision-detected object and the projection of the robotic arm's range of motion.

[0122] In one embodiment, the collision prediction module 30 is further configured to compare the projection of the collision-detected object from the same monitoring camera viewpoint with the projection of the robotic arm's movement range frame by frame to obtain the time node at which the two projections intersect; if the number of monitoring camera viewpoints that intersect at the same time node is greater than or equal to a preset number, it is determined that there is a collision risk within the predicted trajectory duration; if the number of monitoring camera viewpoints that intersect at the same time node is less than the preset number, it is determined that there is no collision risk within the predicted trajectory duration.

[0123] In one embodiment, the image processing module 20 is further configured to acquire robotic arm task action instructions; obtain the spatial motion trajectory of the robotic arm according to the robotic arm task action instructions; decompose the spatial motion trajectory frame by frame to obtain the spatial position of the robotic arm at each moment; acquire the bounding box of the robotic arm from the corresponding viewpoint of each monitoring camera based on the spatial position of the robotic arm; and summarize the bounding boxes of the robotic arm from the same monitoring camera viewpoint to obtain the projection of the robotic arm's motion range.

[0124] In one embodiment, the control module 40 is further configured to send a pause command to the robotic arm and activate a warning light if the collision prediction result indicates a collision risk, until the collision detection target leaves the movement range of the robotic arm; if the collision prediction result indicates no collision risk, the robotic arm continues to perform its predetermined task and continuously monitors the work area to update the collision prediction result in real time.

[0125] This application captures images of the work area using multi-view monitoring cameras, utilizes a deep learning model to identify and track moving objects within the work area, determines potential collision targets, and then predicts the trajectory of the detected targets within a few seconds based on the image bounding boxes over a certain time period. This trajectory is compared with the projected movement range of the robotic arm within those seconds to predict potential collision risks. Control commands are then generated to intelligently control the start and stop states of the robotic arm, ensuring safety in the human-robot collaborative environment. The implementation of this solution significantly improves the safety of human-robot collaboration in industrial environments. Through real-time monitoring and intelligent collision risk prediction, it effectively reduces personnel injuries and equipment damage caused by accidental collisions within the work area, providing an innovative safety protection solution for the field of intelligent manufacturing.

[0126] The planar vision-based workbench safety protection device provided in this application, employing the planar vision-based workbench safety protection method described in the above embodiments, can solve the technical problem of how to effectively predict and prevent collisions between robots and workers. Compared with the prior art, the beneficial effects of the planar vision-based workbench safety protection device provided in this application are the same as those of the planar vision-based workbench safety protection method provided in the above embodiments, and other technical features in the planar vision-based workbench safety protection device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0127] This application provides a workbench safety protection device based on planar vision. The workbench safety protection device based on planar vision includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the workbench safety protection method based on planar vision in the above embodiment 1.

[0128] The following is for reference. Figure 6 This document illustrates a structural schematic diagram of a workbench safety protection device based on planar vision, suitable for implementing embodiments of this application. The workbench safety protection device based on planar vision in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The workbench safety protection device based on planar vision shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0129] like Figure 6As shown, the planar vision-based workbench safety protection device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the planar vision-based workbench safety protection device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the planar vision-based workbench safety device to exchange data wirelessly or via wired communication with other devices. Although planar vision-based workbench safety devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0130] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0131] The planar vision-based workbench safety protection device provided in this application, employing the planar vision-based workbench safety protection method described in the above embodiments, can solve the technical problem of how to effectively predict and prevent collisions between robots and workers. Compared with the prior art, the beneficial effects of the planar vision-based workbench safety protection device provided in this application are the same as those of the planar vision-based workbench safety protection method provided in the above embodiments, and other technical features of this planar vision-based workbench safety protection device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0132] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0133] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0134] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the workbench safety protection method based on planar vision in the above embodiments.

[0135] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0136] The aforementioned computer-readable storage medium may be included in a planar vision-based workbench safety protection device; or it may exist independently and not be assembled into a planar vision-based workbench safety protection device.

[0137] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a planar vision-based workbench safety protection device, cause the planar vision-based workbench safety protection device to: determine collision detection targets within the work area based on multi-view work area images; acquire image bounding boxes of the collision detection targets in the work area images from various viewpoints; obtain collision prediction results based on the projection of the robotic arm's motion range onto the image bounding boxes of the collision detection targets; and generate robotic arm control commands based on the collision prediction results to control the start and stop states of the robotic arm.

[0138] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0140] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0141] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described planar vision-based workbench safety protection method, which can solve the technical problem of how to effectively predict and prevent collisions between robots and workers. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the planar vision-based workbench safety protection method provided in the above embodiments, and will not be repeated here.

[0142] The computer program product provided in this application can solve the technical problem of workbench safety protection based on planar vision. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the workbench safety protection method based on planar vision provided in the above embodiments, and will not be repeated here.

[0143] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A workbench safety protection method based on planar vision, characterized in that, The workbench safety protection method based on planar vision includes: Based on multi-view images of the working area, determine the collision detection targets within the working area; Obtain the image bounding box of the collision detection target in the working area image from various viewpoints; The collision prediction result is obtained by comparing the projection of the robotic arm's range of motion with the image bounding box of the collision detection target. The step of obtaining the collision prediction result based on the image bounding box of the collision detection target and the projection of the robotic arm's range of motion includes: The image bounding boxes of the collision-detected objects within a preset time period are summarized to obtain a set of image bounding box positions. Based on the set of image bounding box positions, the predicted trajectory of the collision-detected object under the viewpoints of each monitoring camera is obtained; Based on the predicted trajectory of the collision-detected object, the projection of the collision-detected object is obtained; The collision prediction result is obtained based on the projection of the collision-detected object and the projection of the robotic arm's range of motion. The step of obtaining the collision prediction result based on the projection of the collision-detected object and the projection of the robotic arm's range of motion includes: The collision detection object projection from the same surveillance camera viewpoint is compared frame by frame with the robotic arm motion range projection to obtain the time point when the two projections intersect. If the number of surveillance camera viewpoints that generate projection intersection at the same time point is greater than or equal to the preset number, then it is determined that there is a collision risk within the predicted trajectory duration. If the number of surveillance camera viewpoints that generate projection intersection at the same time point is less than the preset number, it is determined that there is no risk of collision within the predicted trajectory duration. Based on the collision prediction results, control commands for the robotic arm are generated to control the start and stop states of the robotic arm.

2. The workbench safety protection method based on planar vision according to claim 1, characterized in that, The step of determining the collision detection target within the working area based on multi-view working area images includes: The working area images are acquired from multiple perspectives, and the spatial positions of the multiple monitoring cameras satisfy a surround layout. Based on the moving object detection model, the moving object recognition results are obtained by identifying multiple consecutive frames of working area images from the same viewpoint. The results of moving object recognition from various perspectives are summarized and screened to determine the collision detection targets within the working area.

3. The workbench safety protection method based on planar vision according to claim 2, characterized in that, The step of summarizing and screening the moving object recognition results from various perspectives to determine the collision detection targets within the working area includes: Based on the moving object recognition results from the perspective of each surveillance camera and the spatial position information of the corresponding surveillance camera, the spatial position of each moving object is obtained. If the spatial position of the moving object coincides with the range of the material area, then the object is determined to be an object in the material area. If the spatial position of the moving object coincides with the working area of ​​the robotic arm, then the object is determined to be a robotic arm. By eliminating material objects and robotic arms from the moving object, collision detection targets within the working area are obtained.

4. The workbench safety protection method based on planar vision according to claim 1, characterized in that, Before obtaining the collision prediction result based on the image bounding box of the collision detection target projected from the robot arm's range of motion, the method further includes: Obtain the robotic arm's task motion instructions; The spatial motion trajectory of the robotic arm is obtained according to the task action command of the robotic arm; The spatial motion trajectory is decomposed frame by frame to obtain the spatial position of the robotic arm at each moment. Based on the spatial position of the robotic arm, obtain the bounding box of the robotic arm from the corresponding viewpoint of each monitoring camera; The bounding boxes of the robotic arm from the same surveillance camera perspective are summarized to obtain the projection of the robotic arm's motion range.

5. The workbench safety protection method based on planar vision according to claim 1, characterized in that, The step of generating robotic arm control commands based on the collision prediction results to control the start and stop states of the robotic arm includes: If the collision prediction result indicates a collision risk, a pause command is sent to the robotic arm, and a warning light is activated until the collision detection target leaves the robotic arm's range of motion. If the collision prediction result indicates that there is no collision risk, the robot arm continues to perform its scheduled task while continuously monitoring the work area to update the collision prediction result in real time.

6. A workbench safety protection device based on planar vision, characterized in that, The workbench safety protection device based on planar vision includes: The target recognition module is used to determine the collision detection targets within the working area based on multi-view images of the working area; The image processing module is used to acquire the image bounding box of the collision detection target in the working area image from various viewpoints; The collision prediction module is used to obtain the collision prediction result based on the image bounding box of the robot arm's motion range projection and the collision detection target; The collision prediction module is further configured to summarize the image bounding boxes of the collision-detected object within a preset time period to obtain a set of image bounding box positions; based on the set of image bounding box positions, obtain the predicted trajectory of the collision-detected object from the perspective of each monitoring camera; based on the predicted trajectory of the collision-detected object, obtain the projection of the collision-detected object; and based on the projection of the collision-detected object and the projection of the robotic arm's range of motion, obtain the collision prediction result. The collision prediction module is also used to compare the projection of the collision-detected object from the same monitoring camera view with the projection of the robotic arm's movement range frame by frame to obtain the time node when the two projections intersect; if the number of monitoring camera viewpoints that intersect at the same time node is greater than or equal to a preset number, it is determined that there is a collision risk within the predicted trajectory duration; if the number of monitoring camera viewpoints that intersect at the same time node is less than a preset number, it is determined that there is no collision risk within the predicted trajectory duration. The control module is used to generate control commands for the robotic arm based on the collision prediction results, so as to control the start and stop states of the robotic arm.

7. A workbench safety protection device based on planar vision, characterized in that, The planar vision-based workbench safety protection device includes: a memory, a processor, and a planar vision-based workbench safety protection program stored in the memory and executable on the processor, wherein the planar vision-based workbench safety protection program is configured to implement the steps of the planar vision-based workbench safety protection method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a workbench safety protection program based on planar vision. When the workbench safety protection program based on planar vision is executed by the processor, it implements the steps of the workbench safety protection method based on planar vision as described in any one of claims 1 to 5.

Images