Workbench safety protection method, device and equipment based on plane vision and storage medium

Abstract

The invention relates to the technical field of automation, in particular to a workbench safety protection method, device and equipment based on plane vision and a storage medium. A multi-view monitoring camera is used for capturing a working area image, a deep learning model is used for recognizing and tracking a moving object in a working area, and a detection target possibly generating collision is determined; and then, according to an image bounding box of the detection target within a certain time, predicting a motion trail within a few seconds, comparing the motion trail with a motion range projection of the mechanical arm within a few seconds so as to predict a potential collision risk, and according to the potential collision risk, generating a control instruction to intelligently control the start-stop state of the mechanical arm so as to ensure the safety of a man-machine cooperation environment. The implementation of the scheme obviously improves the safety of man-machine cooperation in the industrial environment, effectively reduces personnel injury and equipment damage caused by accidental collision in a working area through real-time monitoring and intelligent prediction of the collision risk, and provides an innovative safety protection solution for the field of intelligent manufacturing.

Description

Technical Field

[0001] The present invention relates to the field of automation technology, and in particular to a workbench safety protection method, device, equipment and storage medium based on planar vision. Background Art

[0002] As the concept of smart manufacturing is promoted around the world, the manufacturing industry is undergoing a profound technological change. Smart manufacturing systems can monitor production processes in real time, automatically adjust production plans, optimize resource allocation, and improve production efficiency and product quality. Against this backdrop, factories and warehouses are increasingly deploying automated equipment and robots to reduce labor costs, improve production efficiency and flexibility, and meet personalized and customized production needs.

[0003] In the process of advancing intelligent manufacturing, human-machine collaboration has become an important development direction. Human-machine collaboration allows workers and robots to work together in the same work area to give full play to their respective advantages and improve production efficiency and flexibility. However, this collaborative model also brings new safety challenges. Traditional industrial robots are usually isolated in a specific work area to prevent direct contact with workers. But in a human-machine collaborative environment, robots need to operate near workers, which increases the risk of collision between robots and workers.

[0004] Therefore, how to effectively predict and prevent collisions between robots and workers has become an urgent problem to be solved in this field. Summary of the invention

[0005] The main purpose of the present invention is to provide a workbench safety protection method, device, equipment and storage medium based on planar vision, aiming to solve the technical problem of how to effectively predict and prevent collisions between robots and workers in the prior art.

[0006] To achieve the above object, the present invention provides a workbench safety protection method based on planar vision, the method comprising the following steps: According to the multi-view working area images, the collision detection targets in the working area are determined; Obtaining an image bounding box of the collision detection target in an image of a working area at each viewing angle; Obtaining a collision prediction result based on the projection of the robot arm's motion range and the image bounding box of the collision detection target; According to the collision prediction result, a robot arm control instruction is generated to control the start and stop state of the robot arm.

[0007] Optionally, determining the collision detection target in the working area according to the multi-view working area image includes: Acquire images of the working area from multiple viewing angles, wherein the spatial positions of the multiple monitoring cameras satisfy a surround layout; Based on the moving object detection model, multiple consecutive frames of working area images under the same viewing angle are identified to obtain the moving object recognition result; The moving object recognition results under various viewing angles are summarized and screened to determine the collision detection targets within the working area.

[0008] Optionally, the summarizing and screening the moving object recognition results at various viewing angles to determine the collision detection target within the working area includes: According to the recognition results of moving objects under the viewing angles of each monitoring camera and the spatial position information of the corresponding monitoring camera, the spatial position of each moving object is obtained; If the spatial position of the moving object coincides with the range of the material area, the object is determined to be a material area object; If the spatial position of the moving object coincides with the working area of ​​the robotic arm, the object is determined to be a robotic arm; Objects in the material area and the robot arm are screened out from the moving objects to obtain collision detection targets in the working area.

[0009] Optionally, obtaining a collision prediction result based on the projection of the robot arm motion range and the image bounding box of the collision detection target includes: Summarizing the image bounding boxes of the collision detection objects within a preset time period to obtain an image bounding box position set; Obtaining predicted trajectories of the collision detection object under the viewing angles of each surveillance camera according to the image bounding box position set; Obtaining a projection of the collision detection object according to the predicted trajectory of the collision detection object; A collision prediction result is obtained according to the projection of the collision detection object and the projection of the motion range of the robot arm.

[0010] Optionally, obtaining a collision prediction result according to the projection of the collision detection object and the projection of the motion range of the robot arm includes: Compare the projection of the collision detection object under the same monitoring camera's viewing angle with the projection of the robot arm's motion range frame by frame to obtain the time node where the two projections intersect; If the number of surveillance camera views that generate projection intersections at the same time node is greater than or equal to the preset number, it is determined that there is a collision risk within the predicted trajectory duration; If the number of surveillance camera views that produce projection intersection at the same time node is less than the preset number, it is judged that there is no collision risk within the predicted trajectory duration.

[0011] Optionally, before obtaining the collision prediction result based on the projection of the image bounding box of the collision detection target with the motion range of the robot arm, the method further includes: Get the robot arm task action instructions; According to the task action instruction of the robot arm, a spatial motion trajectory of the robot arm is obtained; Decomposing the spatial motion trajectory frame by frame to obtain the spatial position of the robot arm at each moment; Based on the spatial position of the robotic arm, obtaining the robotic arm boundary box of each surveillance camera at its corresponding viewing angle; The robot arm bounding boxes under the same monitoring camera perspective are summarized to obtain the robot arm motion range projection.

[0012] Optionally, generating a robot arm control instruction according to the collision prediction result to control the start and stop state of the robot arm includes: If the collision prediction result indicates that there is a collision risk, a pause instruction is sent to the robot arm, and a warning indicator light is activated at the same time until the collision detection target leaves the motion range of the robot arm; If the collision prediction result indicates that there is no collision risk, the robot continues to perform the scheduled task and continuously monitors the working area to update the collision prediction result in real time.

[0013] In addition, to achieve the above-mentioned purpose, the present invention also proposes a workbench safety protection device based on plane vision, and the workbench safety protection device based on plane vision includes: A target recognition module is used to determine the collision detection target in the working area according to the multi-view working area image; An image processing module, used to obtain an image boundary box of the collision detection target in the working area image at each viewing angle; A collision prediction module, used to obtain a collision prediction result according to the image boundary box of the robot arm motion range projection and the collision detection target; The control module is used to generate a robot arm control instruction according to the collision prediction result to control the start and stop state of the robot arm.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also proposes a workbench safety protection device based on plane vision, and the workbench safety protection device based on plane vision includes: a memory, a processor, and a workbench safety protection program based on plane vision stored in the memory and executable on the processor, and the workbench safety protection program based on plane vision is configured to implement the steps of the workbench safety protection method based on plane vision as described above.

[0015] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a workbench safety protection program based on plane vision is stored. When the workbench safety protection program based on plane vision is executed by a processor, the steps of the workbench safety protection method based on plane vision as described above are implemented.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: This application captures images of the working area through a multi-view surveillance camera, uses a deep learning model to identify and track moving objects in the working area, determines the detection targets that may cause collisions, and then predicts the motion trajectory within a few seconds based on the image boundary box of the detection target within a certain period of time, and compares it with the projection of the motion range of the robot arm within a few seconds, thereby predicting potential collision risks, and generating control instructions based on this to intelligently control the start and stop status of the robot arm to ensure the safety of the human-machine collaborative environment. The implementation of this solution significantly improves the safety of human-machine collaboration in industrial environments. Through real-time monitoring and intelligent prediction of collision risks, it effectively reduces personal injuries and equipment damage caused by accidental collisions in the working area, providing an innovative safety protection solution for the field of intelligent manufacturing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a flow chart of the first embodiment of the workbench safety protection method based on plane vision of the present invention; Figure 2 It is a flow chart of a second embodiment of a workbench safety protection method based on plane vision of the present invention; Figure 3 A schematic diagram of a human-machine collaborative workbench according to a workbench safety protection method based on planar vision of the present invention; Figure 4 It is a flow chart of a third embodiment of a workbench safety protection method based on plane vision of the present invention; Figure 5 It is a structural block diagram of the first embodiment of the workbench safety protection device based on plane vision of the present invention; Figure 6 It is a structural schematic diagram of a workbench safety protection device based on plane vision in a hardware operating environment involved in an embodiment of the present invention.

[0020] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0022] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of the embodiment of the present application is: determine the collision detection target in the working area based on the multi-view working area image; obtain the image boundary box of the collision detection target in the working area image at each view angle; obtain the collision prediction result based on the projection of the robot arm motion range and the image boundary box of the collision detection target; generate the robot arm control instruction based on the collision prediction result to control the start and stop state of the robot arm.

[0024] Currently, some human-machine collaborative automation workbenches are equipped with robotic arms. Workers work together with mechanical equipment on these workbenches. Therefore, it is necessary to ensure that the robotic arms do not collide with workers' hands or other products or components during normal operation.

[0025] Traditional technologies usually set up physical barriers or protective fences around the robot arm to separate the robot arm's working area from the worker's activity area, or use safety gratings (safety light curtains) to monitor the robot arm's working area. When the grating is blocked, the robot arm will automatically stop moving, but the safety grating can only be set in a fixed area, which is very difficult to implement when different protective measures are needed for different work contents. These traditional technologies have improved the safety of the human-machine collaborative environment to a certain extent, but they usually have certain limitations, such as physical isolation limits flexibility, sensors may have blind spots, and most of them rely on passive protection rather than active prediction and avoidance of collisions.

[0026] This application provides a solution that uses a multi-view surveillance camera to capture images of the work area, uses a deep learning model to identify and track moving objects in the work area, determines the detection targets that may cause collisions, and then predicts the motion trajectory within a few seconds based on the image boundary box of the detection target within a certain period of time, and compares it with the projection of the motion range of the robot arm within a few seconds, thereby predicting potential collision risks, and generating control instructions based on this to intelligently control the start and stop status of the robot arm to ensure the safety of the human-machine collaborative environment. The implementation of this solution significantly improves the safety of human-machine collaboration in industrial environments. Through real-time monitoring and intelligent prediction of collision risks, it effectively reduces personal injuries and equipment damage caused by accidental collisions in the work area, providing an innovative safety protection solution for the field of intelligent manufacturing.

[0027] Based on this, the embodiment of the present invention provides a workbench safety protection method based on plane vision, referring to Figure 1 , Figure 1 It is a flow chart of a first embodiment of a workbench safety protection method based on planar vision of the present invention.

[0028] In this embodiment, the workbench safety protection method based on plane vision includes the following steps: Step S10: Determine the collision detection target in the working area according to the multi-view working area image.

[0029] It should be noted that the purpose of this step is to use multiple surveillance cameras at different locations to collect image data of the same working area, and to identify and locate objects in the working area that may collide with the robotic arm based on the captured image data.

[0030] It is understandable that at the same time, surveillance cameras with different perspectives may only capture part of the images that may collide with the robotic arm. In order to avoid missing or misjudging objects that may collide with the robotic arm, multiple surveillance cameras are arranged in a surround manner on the material workbench to ensure that every corner of the workbench is within the monitoring range of at least one camera. There are overlapping areas of perspective between adjacent surveillance cameras, which helps to improve the accuracy of object detection and reduce misjudgment through cross-validation of multiple perspectives.

[0031] It should be understood that determining the collision detection target is not just about detecting the existence of an object, but also includes dynamic tracking and identification of the object. The movement of any object on the workbench can be continuously tracked by multiple cameras, and tracking can be maintained even if the object moves quickly or changes position on the workbench. In addition, due to the characteristics of the workbench, in addition to the operator's hands and robotic arms, there may be some workpieces on the material table itself, and their positions may move as the engineering work progresses. Therefore, in the process of identifying the collision detection target, it is also necessary to exclude objects in the material area and the robotic arm itself, and only focus on other objects that may collide with the robotic arm.

[0032] Step S20: obtaining an image boundary box of the collision detection target in the working area image at each viewing angle.

[0033] It should be noted that the present invention adopts the YOLOv8 network model as the main framework of the detection model, and performs recognition and classification training on objects to be detected that may enter the device before the workbench is officially run. It can be understood that the YOLOv8 network model is an advanced deep learning technology specifically used for real-time object detection tasks. It can quickly and accurately identify and classify objects in images and provide precise bounding boxes of objects.

[0034] It should be understood that as time goes by, the bounding box formed by the collision detection target at each viewing angle will move, and this movement may approach the working swing area of ​​the robot arm or may leave the workbench.

[0035] Step S30: Obtain a collision prediction result based on the projection of the robot arm's motion range and the image boundary box of the collision detection target.

[0036] It should be noted that based on the movement of the collision detection target in the past period of time, the possible movement trajectory in a shorter period of time can be roughly estimated. Generally, statistical or machine learning methods, such as linear regression, Gaussian process or more complex deep learning models, can be used to predict the future position of the object.

[0037] It can be understood that the Kalman filter algorithm is used in the present application to estimate the dynamic state of the object, including position and speed, and predict the state at the next moment. In the specific implementation process, the monitoring camera collects images at a speed of 30 frames or 60 frames per second. The system predicts the position of the image boundary box of the collision detection target within the next 2 seconds based on the multiple frames of images taken in the past 3 seconds. Generally speaking, the prediction time of 2 seconds here is based on practical application considerations. This time window is neither too long to make it difficult to accurately predict the rapidly changing dynamic environment, nor too short to be insufficient to take effective preventive measures.

[0038] It should be understood that as time goes by, the multiple frames of images of the collision detection target in the past 3 seconds will change over time, and the corresponding predicted trajectory will also change. Only when the predicted trajectory overlaps with the projection of the robot arm's motion range will it be considered that there is a collision risk.

[0039] In one embodiment, before obtaining the collision prediction result based on the projection of the robot arm's motion range and the image bounding box of the collision detection target, the method further includes: obtaining a robot arm task action instruction; obtaining the spatial motion trajectory of the robot arm according to the robot arm task action instruction; decomposing the spatial motion trajectory frame by frame to obtain the spatial position of the robot arm at each moment; based on the spatial position of the robot arm, obtaining the robot arm bounding box of each monitoring camera at its corresponding viewing angle; and summarizing the robot arm bounding boxes at the same monitoring camera viewing angle to obtain the projection of the robot arm's motion range.

[0040] It should be noted that the process and results of mapping the motion trajectory or working area of ​​the robot in three-dimensional space onto a two-dimensional plane (usually the viewing plane of the surveillance camera) are very important for safety monitoring and collision prevention systems. This projection allows us to visualize and analyze the movement of the robot on a plane.

[0041] It is understandable that since the robotic arm operates according to fixed task action instructions, the trajectory it generates is predictable, that is, the movement of the robotic arm within a period of time can be obtained according to the task action instructions at any time. Based on this, the movement range of the robotic arm under the perspective of each monitoring camera can be obtained, and the minimum bounding box surrounding this range can be determined.

[0042] It should be understood that when the predicted trajectory of the collision detection target at a certain moment indicates that it will overlap with the bounding box in the projection of the robot's motion range within the predicted time, then it should be considered that there is a possibility of collision within this time period. On the other hand, in order to reduce the need for false detection, it will also query whether there is a bounding box overlap at other perspectives at the same time. Only when the number reaches the confidence standard will it be considered that there is a collision risk.

[0043] Step S40: generating a robot arm control instruction according to the collision prediction result to control the start and stop state of the robot arm.

[0044] It should be noted that if a collision is predicted, the robot arm’s action will be suspended until the detection target leaves the robot arm and reaches a safe position before continuing the previous work.

[0045] In one embodiment, a robot arm control instruction is generated based on the collision prediction result to control the start and stop state of the robot arm, including: if the collision prediction result indicates that there is a collision risk, a pause instruction is sent to the robot arm, and a warning prompt light is activated until the collision detection target leaves the motion range of the robot arm; if the collision prediction result indicates that there is no collision risk, the robot arm continues to perform the scheduled task, and continuously monitors the working area to update the collision prediction result in real time.

[0046] Understandably, pausing the robot arm’s action and lighting up the indicator light is a preventive measure designed to provide workers with an obvious warning and ensure that they have enough time and space to deal with potential dangers. Based on this control logic, the robot arm can stop the action in time to ensure the safety of workers and equipment, and can quickly resume normal work after the risk is eliminated, reducing production interruptions. This dynamic safety control strategy is the key to achieving efficient human-machine collaboration.

[0047] This embodiment determines the collision detection target in the working area based on the multi-view working area image; obtains the image boundary box of the collision detection target in the working area image at each view angle; obtains the collision prediction result based on the projection of the robot arm motion range and the image boundary box of the collision detection target; and generates the robot arm control instruction based on the collision prediction result to control the start and stop state of the robot arm.

[0048] In summary, in this embodiment, the image of the working area is captured by a multi-view surveillance camera, and the moving objects in the working area are identified and tracked using a deep learning model to determine the detection target that may cause a collision. Then, the motion trajectory within a few seconds is predicted based on the image boundary box of the detection target within a certain period of time, and compared with the projection of the motion range of the robot arm within a few seconds, so as to predict the potential collision risk, and generate control instructions accordingly to intelligently control the start and stop status of the robot arm to ensure the safety of the human-machine collaborative environment. The implementation of this solution significantly improves the safety of human-machine collaboration in industrial environments, and effectively reduces personal injuries and equipment damage caused by accidental collisions in the working area through real-time monitoring and intelligent prediction of collision risks.

[0049] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 , the step S10 further includes: Step S101: Acquire images of the working area under multiple viewing angles, wherein the spatial positions of the multiple monitoring cameras satisfy a surround layout.

[0050] like Figure 3 As shown, Figure 3It is a schematic diagram of the human-machine collaborative workbench in the present invention, which includes the following components: an overall frame 1 of the device, which is used to maintain the stability of the device as a whole, a multi-joint collaborative manipulator 2 inverted on the frame 1, the manipulator ensures the stability of the overall frame during movement, a working area 3, which is arranged in the frame 1 to ensure that the area is within the travel range of the manipulator 2, and other different auxiliary production structures can be installed as needed, and a number of security monitoring cameras 4 installed at different positions and angles on the frame 1.

[0051] Step S102: Based on the moving object detection model, multiple frames of continuous working area images under the same viewing angle are identified to obtain a moving object recognition result.

[0052] It should be noted that by analyzing continuous image sequences from the same perspective through a moving object detection model, moving objects in the working area can be identified and tracked. This process usually involves separating dynamic objects from the static background and determining their motion characteristics, such as their position, size and possible direction of movement in the image.

[0053] It is understandable that even if the moving objects identified at different viewing angles appear different, they may actually be the same object. Therefore, a data association process is required to match and associate the objects identified at different viewing angles to obtain a complete understanding of each independent object. For example, if one of the cameras identifies three moving objects and the surveillance camera at another viewing angle identifies four moving objects, the position of the moving objects in the image and the position of the surveillance camera in the workbench area can be used to determine whether there is overlap between these moving objects.

[0054] Step S103: Summarize and screen the moving object recognition results at various viewing angles to determine the collision detection target within the working area.

[0055] It should be noted that under normal circumstances, the system will integrate the results of moving object recognition from multiple perspectives to build a comprehensive and dynamically updated view of the work area. This includes tracking and identifying all potential moving objects, that is, the recognition process here includes the operator's limbs, the robot arm, and the workpiece on the material table.

[0056] It is understandable that due to the characteristics of the material table, the vertical height of the object on the material table is fixed. Therefore, the position of the moving object in the image under multiple perspectives can be used to roughly determine whether the object is a workpiece on the material table. If so, subsequent collision prediction and position tracking are not required.

[0057] In one embodiment, the moving object recognition results under various viewing angles are summarized and screened to determine the collision detection targets in the working area, including: obtaining the spatial position of each moving object based on the moving object recognition results under the viewing angles of each monitoring camera and the spatial position information of the corresponding monitoring camera; if the spatial position of the moving object coincides with the material area range, then the object is judged to be a material area object; if the spatial position of the moving object coincides with the working area of ​​the robotic arm, then the object is judged to be a robotic arm; and the material area objects and robotic arms among the moving objects are screened out to obtain the collision detection targets in the working area.

[0058] It should be noted that by integrating the moving object recognition results from multiple perspectives and combining them with the spatial position information of the monitoring camera, the exact position of the object in the working area can be determined. Then, objects in the material area and the robotic arm are screened out to avoid false detection, because these objects are usually known, fixed or controlled. The targets that need to be collided with can be determined through the above screening process.

[0059] This embodiment obtains images of the working area from multiple perspectives, and the spatial positions of the multiple surveillance cameras satisfy a surround layout; based on a moving object detection model, multiple consecutive frames of working area images from the same perspective are identified to obtain moving object recognition results; the moving object recognition results from each perspective are summarized and screened to determine collision detection targets within the working area.

[0060] In summary, in this embodiment, the accuracy and efficiency of collision detection are significantly improved through precise spatial position determination and intelligent screening mechanism. It not only reduces false detection and unnecessary safety intervention, but also ensures the safety and reliability of robot arm operation in a dynamically changing working environment. In addition, the real-time monitoring and dynamic update capabilities of this solution enable the safety protection system to respond quickly to emergencies and take timely measures, thereby reducing safety risks in the working area, protecting the safety of workers and equipment, and improving production efficiency.

[0061] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 , step S30, comprising: Step S301: Summarize the image boundary boxes of the collision detection object within a preset time period to obtain an image boundary box position set.

[0062] It should be noted that the preset time period here refers to the period from the current moment to the previous few seconds. The span of this time window is related to the accuracy requirement of trajectory prediction.

[0063] It is understandable that the surveillance camera will capture multiple images within a few seconds, and the position change of the collision detection object in the image will form a continuous position change track.

[0064] Step S302: obtaining the predicted trajectory of the collision detection object under the viewing angle of each monitoring camera according to the image bounding box position set.

[0065] It should be noted that this step involves the data collected by multiple surveillance cameras. Each camera may provide different perspectives and information, so these data need to be integrated to obtain a comprehensive view of the object's movement.

[0066] It is understandable that since the position of an object changes under different camera perspectives, this information can be used to construct the 3D motion trajectory of the object. This usually involves the transformation of spatial coordinates and data fusion technology to ensure that the data obtained from different perspectives can be accurately mapped into the same 3D space.

[0067] It should be understood that the accuracy of the predicted trajectory depends not only on the quality of the data from a single camera, but also on the integration of data from multiple cameras. This means that even if the data from a single camera is very accurate, if the data from different cameras is not integrated properly, the final predicted trajectory may be inaccurate. Therefore, this step requires highly precise data processing and analysis techniques to ensure that the data obtained from multiple perspectives can be correctly integrated and interpreted.

[0068] Step S303: obtaining a projection of the collision detection object according to the predicted trajectory of the collision detection object.

[0069] It should be noted that, since the predicted trajectory is a series of position changes over time, the object position in each frame can be extracted and mapped to a common reference plane to form a projection of the collision detection object in the viewing angle of the monitoring camera.

[0070] It is understandable that, since there are multiple surveillance cameras at the same time, it is necessary to aggregate the projections of collision detection objects under multiple surveillance perspectives at the same time, which can improve the accuracy of the information source of subsequent collision detection object projections used for target detection and tracking.

[0071] Step S304: Obtaining a collision prediction result according to the projection of the collision detection object and the projection of the motion range of the robot arm.

[0072] In one embodiment, the collision prediction result is obtained based on the projection of the collision detection object and the projection of the motion range of the robotic arm, including: comparing the projection of the collision detection object and the projection of the motion range of the robotic arm frame by frame under the same monitoring camera perspective to obtain the time node at which the projections of both parties intersect; if the number of monitoring camera perspectives that produce projection intersections at the same time node is greater than or equal to a preset number, it is determined that there is a collision risk within the predicted trajectory duration; if the number of monitoring camera perspectives that produce projection intersections at the same time node is less than a preset number, it is determined that there is no collision risk within the predicted trajectory duration.

[0073] It should be noted that the projection of the object and the projection of the robotic arm are compared at the same time point to determine whether there is a potential collision. This process requires precise time synchronization and spatial alignment to ensure the accuracy of the comparison. By comparing the projection of the object and the projection of the robotic arm from the same surveillance camera perspective frame by frame, the relative position of the two in space can be determined. This method can evaluate the possibility of collision at each time point in the time period of the predicted trajectory and make decisions accordingly.

[0074] It is understandable that the preset number is a threshold used to determine whether there is sufficient evidence to indicate a risk of collision. If multiple perspectives show an intersection at the same time, the possibility of a collision can be predicted with greater confidence. This method uses information from multiple perspectives to enhance the reliability of the prediction. For example, when there are 4 surveillance cameras in the system at the same time, at least 3 surveillance cameras are required to detect the intersection of the projections of both parties before a comprehensive conclusion that there is a risk of collision is reached.

[0075] This embodiment constructs the three-dimensional motion trajectory of the object by summarizing the bounding boxes of the collision detection object images captured by multiple surveillance cameras within a preset time period, and projects it onto a two-dimensional plane. By comparing the object projection with the robot arm motion range projection frame by frame, the system can predict potential collision risks. Specifically, the system compares the object projection under the same surveillance camera perspective with the robot arm projection, determines the time node where the intersection occurs, and determines whether there is a collision risk based on the number of surveillance camera perspectives that produce the intersection. This method improves the accuracy and reliability of collision detection because it integrates information from multiple perspectives and reduces the errors and uncertainties that may be caused by a single perspective.

[0076] In summary, in this embodiment, by integrating the data of multiple surveillance cameras, comprehensive monitoring and accurate prediction of the object's motion trajectory are achieved, thereby significantly improving the accuracy of collision detection. This method can not only reduce false alarms and missed alarms, but also respond in real time in a dynamic environment, providing safety guarantees for the operation of the robotic arm. By setting a threshold to judge the risk of collision, the system can issue a warning only when multiple perspectives consistently show potential collisions, which enhances the credibility of the prediction results. In addition, the solution can also adapt to different surveillance camera layouts and environmental conditions, has good flexibility and adaptability, and provides strong support for the safe operation of automation and robotic systems.

[0077] This application also provides a workbench safety protection device based on plane vision, please refer to Figure 5 , the workbench safety protection device based on plane vision includes: The target recognition module 10 is used to determine the collision detection target in the working area according to the multi-view working area image; An image processing module 20 is used to obtain an image boundary box of the collision detection target in the working area image at each viewing angle; A collision prediction module 30, configured to obtain a collision prediction result according to the image boundary box of the robot arm motion range projection and the collision detection target; The control module 40 is used to generate a robot arm control instruction according to the collision prediction result to control the start and stop state of the robot arm.

[0078] In one embodiment, the target recognition module 10 is also used to obtain images of the working area from multiple perspectives, and the spatial positions of the multiple surveillance cameras satisfy a surround layout; based on a moving object detection model, multiple consecutive frames of working area images from the same perspective are recognized to obtain moving object recognition results; the moving object recognition results from each perspective are summarized and screened to determine the collision detection target within the working area.

[0079] In one embodiment, the target recognition module 10 is also used to obtain the spatial position of each moving object based on the moving object recognition results under the viewing angle of each monitoring camera and the spatial position information of the corresponding monitoring camera; if the spatial position of the moving object coincides with the material area range, the object is judged to be a material area object; if the spatial position of the moving object coincides with the working area of ​​the robotic arm, the object is judged to be a robotic arm; the material area objects and robotic arms in the moving objects are screened out to obtain the collision detection target within the working area.

[0080] In one embodiment, the collision prediction module 30 is further used to summarize the image bounding boxes of the collision detection object within a preset time period to obtain an image bounding box position set; based on the image bounding box position set, obtain the predicted trajectory of the collision detection object under the viewing angle of each monitoring camera; based on the predicted trajectory of the collision detection object, obtain the projection of the collision detection object; based on the projection of the collision detection object and the projection of the motion range of the robotic arm, obtain a collision prediction result.

[0081] In one embodiment, the collision prediction module 30 is further used to compare the projection of the collision detection object under the same monitoring camera perspective with the projection of the motion range of the robotic arm frame by frame to obtain the time node at which the projections of both parties intersect; if the number of monitoring camera perspectives that produce projection intersections at the same time node is greater than or equal to a preset number, it is judged that there is a collision risk within the predicted trajectory duration; if the number of monitoring camera perspectives that produce projection intersections at the same time node is less than a preset number, it is judged that there is no collision risk within the predicted trajectory duration.

[0082] In one embodiment, the image processing module 20 is also used to obtain robot arm task action instructions; obtain the spatial motion trajectory of the robot arm according to the robot arm task action instructions; decompose the spatial motion trajectory frame by frame to obtain the spatial position of the robot arm at each moment; based on the spatial position of the robot arm, obtain the robot arm boundary box of each monitoring camera under its corresponding perspective; summarize the robot arm boundary boxes under the same monitoring camera perspective to obtain the projection of the robot arm motion range.

[0083] In one embodiment, the control module 40 is also used to send a pause command to the robotic arm and activate the early warning light if the collision prediction result indicates that there is a collision risk, until the collision detection target leaves the movement range of the robotic arm; if the collision prediction result indicates that there is no collision risk, continue to execute the predetermined task of the robotic arm, and continuously monitor the working area to update the collision prediction result in real time.

[0084] This application uses a multi-view surveillance camera to capture images of the work area, uses a deep learning model to identify and track moving objects in the work area, determines the detection targets that may cause collisions, and then predicts the motion trajectory within a few seconds based on the image bounding box of the detection target within a certain period of time, and compares it with the projection of the robot's motion range within a few seconds, thereby predicting potential collision risks, and generating control instructions based on this to intelligently control the start and stop status of the robot arm to ensure the safety of the human-machine collaborative environment. The implementation of this solution significantly improves the safety of human-machine collaboration in industrial environments. Through real-time monitoring and intelligent prediction of collision risks, it effectively reduces personal injuries and equipment damage caused by accidental collisions in the work area, providing an innovative safety protection solution for the field of intelligent manufacturing.

[0085] The workbench safety protection device based on plane vision provided by the present application adopts the workbench safety protection method based on plane vision in the above embodiment, which can solve the technical problem of how to effectively predict and prevent collisions between robots and workers. Compared with the prior art, the beneficial effects of the workbench safety protection device based on plane vision provided by the present application are the same as the beneficial effects of the workbench safety protection method based on plane vision provided by the above embodiment, and the other technical features of the workbench safety protection device based on plane vision are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0086] The present application provides a workbench safety protection device based on plane vision, and the workbench safety protection device based on plane vision includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the workbench safety protection method based on plane vision in the above-mentioned embodiment one.

[0087] Reference below Figure 6 , which shows a schematic diagram of the structure of a workbench safety protection device based on plane vision suitable for implementing the embodiment of the present application. The workbench safety protection device based on plane vision in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The workbench safety protection equipment based on planar vision shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0088] like Figure 6As shown, the workbench safety protection equipment based on plane vision may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the workbench safety protection equipment based on plane vision are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the planar vision-based workbench safety protection equipment to communicate wirelessly or wired with other devices to exchange data. Although the planar vision-based workbench safety protection equipment with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0089] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0090] The workbench safety protection equipment based on plane vision provided by the present application adopts the workbench safety protection method based on plane vision in the above embodiment, which can solve the technical problem of how to effectively predict and prevent collisions between robots and workers. Compared with the prior art, the beneficial effects of the workbench safety protection equipment based on plane vision provided by the present application are the same as the beneficial effects of the workbench safety protection method based on plane vision provided by the above embodiment, and the other technical features of the workbench safety protection equipment based on plane vision are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0091] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0092] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0093] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the planar vision-based workbench safety protection method in the above-mentioned embodiment.

[0094] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0095] The above-mentioned computer-readable storage medium may be included in the workbench safety protection device based on plane vision; or it may exist independently without being assembled into the workbench safety protection device based on plane vision.

[0096] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the workbench safety protection equipment based on plane vision, the workbench safety protection equipment based on plane vision enables: to determine the collision detection target in the working area according to the multi-perspective working area image; to obtain the image bounding box of the collision detection target in the working area image at each perspective; to obtain the collision prediction result according to the projection of the robot arm's motion range and the image bounding box of the collision detection target; and to generate the robot arm control instruction according to the collision prediction result to control the start and stop state of the robot arm.

[0097] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0100] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned workbench safety protection method based on plane vision, and can solve the technical problem of how to effectively predict and prevent collisions between robots and workers. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the workbench safety protection method based on plane vision provided in the above-mentioned embodiment, and will not be repeated here.

[0101] The computer program product provided in this application can solve the technical problem of workbench safety protection based on plane vision. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the workbench safety protection method based on plane vision provided in the above embodiment, which will not be repeated here.

[0102] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A workbench safety protection method based on plane vision, characterized in that: The workbench safety protection method based on plane vision includes: According to the multi-view working area images, the collision detection targets in the working area are determined; Obtaining an image bounding box of the collision detection target in an image of a working area at each viewing angle; Obtaining a collision prediction result based on the projection of the robot arm's motion range and the image bounding box of the collision detection target; According to the collision prediction result, a robot arm control instruction is generated to control the start and stop state of the robot arm.

2. The workbench safety protection method based on plane vision according to claim 1 is characterized in that: The step of determining the collision detection target in the working area according to the multi-view working area image includes: Acquire images of the working area from multiple viewing angles, wherein the spatial positions of the multiple monitoring cameras satisfy a surround layout; Based on the moving object detection model, multiple consecutive frames of working area images under the same viewing angle are identified to obtain the moving object recognition result; The moving object recognition results under various viewing angles are summarized and screened to determine the collision detection targets within the working area.

3. The workbench safety protection method based on plane vision according to claim 2 is characterized in that: The method of summarizing and screening the moving object recognition results under various viewing angles to determine the collision detection target within the working area includes: According to the recognition results of moving objects under the viewing angles of each monitoring camera and the spatial position information of the corresponding monitoring camera, the spatial position of each moving object is obtained; If the spatial position of the moving object coincides with the range of the material area, the object is determined to be a material area object; If the spatial position of the moving object coincides with the working area of ​​the robotic arm, the object is determined to be a robotic arm; Objects in the material area and the robot arm are screened out from the moving objects to obtain collision detection targets in the working area.

4. The workbench safety protection method based on plane vision according to claim 1 is characterized in that: The obtaining of the collision prediction result based on the projection of the robot arm motion range and the image boundary box of the collision detection target comprises: Summarizing the image bounding boxes of the collision detection objects within a preset time period to obtain an image bounding box position set; Obtaining predicted trajectories of the collision detection object under the viewing angles of each surveillance camera according to the image bounding box position set; Obtaining a projection of the collision detection object according to the predicted trajectory of the collision detection object; A collision prediction result is obtained according to the projection of the collision detection object and the projection of the motion range of the robot arm.

5. The workbench safety protection method based on plane vision according to claim 4 is characterized in that: Obtaining a collision prediction result according to the projection of the collision detection object and the projection of the motion range of the robot arm includes: Compare the projection of the collision detection object under the same monitoring camera's viewing angle with the projection of the robot arm's motion range frame by frame to obtain the time node where the two projections intersect; If the number of surveillance camera views that generate projection intersections at the same time node is greater than or equal to the preset number, it is determined that there is a collision risk within the predicted trajectory duration; If the number of surveillance camera views that produce projection intersection at the same time node is less than the preset number, it is judged that there is no collision risk within the predicted trajectory duration.

6. The workbench safety protection method based on plane vision according to claim 1 is characterized in that: Before obtaining the collision prediction result based on the image boundary box of the collision detection target projected on the robot arm motion range, the method further includes: Get the robot arm task action instructions; According to the task action instruction of the robot arm, a spatial motion trajectory of the robot arm is obtained; Decomposing the spatial motion trajectory frame by frame to obtain the spatial position of the robot arm at each moment; Based on the spatial position of the robotic arm, obtaining the robotic arm boundary box of each surveillance camera at its corresponding viewing angle; The robot arm bounding boxes under the same monitoring camera perspective are summarized to obtain the robot arm motion range projection.

7. The workbench safety protection method based on plane vision according to claim 1 is characterized in that: Generating a robot arm control instruction according to the collision prediction result to control the start and stop state of the robot arm includes: If the collision prediction result indicates that there is a collision risk, a pause instruction is sent to the robot arm, and a warning indicator light is activated at the same time until the collision detection target leaves the motion range of the robot arm; If the collision prediction result indicates that there is no collision risk, the robot continues to perform the scheduled task and continuously monitors the working area to update the collision prediction result in real time.

8. A workbench safety protection device based on plane vision, characterized in that: The workbench safety protection device based on plane vision includes: A target recognition module is used to determine the collision detection target in the working area according to the multi-view working area image; An image processing module, used to obtain an image boundary box of the collision detection target in the working area image at each viewing angle; A collision prediction module, used to obtain a collision prediction result according to the image boundary box of the robot arm motion range projection and the collision detection target; The control module is used to generate a robot arm control instruction according to the collision prediction result to control the start and stop state of the robot arm.

9. A workbench safety protection device based on plane vision, characterized in that: The plane vision-based workbench safety protection equipment includes: a memory, a processor, and a plane vision-based workbench safety protection program stored in the memory and executable on the processor. The plane vision-based workbench safety protection program is configured to implement the steps of the plane vision-based workbench safety protection method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a workbench safety protection program based on plane vision, and when the workbench safety protection program based on plane vision is executed by the processor, the steps of the workbench safety protection method based on plane vision as described in any one of claims 1 to 7 are implemented.

Images