A method and device for protecting operation safety
The method uses binocular and monocular image frames with neural network correction to enhance safety by reducing false and missed detections, improving positioning accuracy, and ensuring the safety of local body parts in industrial operations.
Patent Information
- Application Number
- CN202411045365.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-01
AI Technical Summary
Existing safety protection methods for industrial operations, such as those involving robotic arms, suffer from limitations and safety risks due to missed or false detections, particularly in complex environments where personnel can enter danger zones unnoticed, leading to potential accidents.
A method utilizing binocular and monocular image frames for target detection, tracking, and disparity estimation to generate a target disparity map, followed by trajectory analysis and security risk assessment, with online correction of false detections using a trained neural network model, to improve safety assurance in the workspace.
Enhances safety by reducing false and missed detections, improving positioning accuracy, and ensuring the safety of local body parts, thereby preventing accidents and maintaining production efficiency.
Smart Images

Figure CN118570260B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of safety protection, and in particular, to a method for protecting operation safety. Background Art
[0002] In industrial production, operation safety protection is one of the effective means to reduce safety accidents. For example, heavy machinery operations represented by industrial robotic arms have strict safety protection measures to prevent personnel from entering when the industrial robotic arms are in operation and causing personal safety accidents.
[0003] Safety accidents can be divided into two categories according to the time of occurrence. One is the danger caused by personnel entering the equipment during operation without stopping the equipment, and the other is the danger caused by not checking whether there are personnel within the safety range when the equipment is started. Common protective devices include isolation fences and interlocking protective devices. Common protective devices include interlocking devices, sensitive protective equipment, active photoelectric protective devices, etc.; however, all of the above methods have certain limitations and safety risks. Summary of the invention
[0004] The present invention provides a method for protecting operation safety, so as to improve the safety guarantee in the operation space.
[0005] The first aspect of the present application provides a method for protecting operation safety, the method comprising:
[0006] Acquire image data in the workspace, wherein the image data includes a binocular image frame,
[0007] Based on any monocular image frame in the image data, target detection is performed with the human body as the target, and the detected target is tracked to obtain target trajectory data, and image segmentation is performed with the entire human body and / or a part of the human body as the target to obtain the segmented target image.
[0008] Using the disparity estimation between binocular image frames in the image data, the principal component analysis of the target point cloud is performed on the segmented target image to obtain the target disparity map.
[0009] Based on the target disparity map, the spatial position information of the target is obtained, and according to the spatial position information of the target, the spatial position of the target trajectory in the target trajectory data is determined,
[0010] Perform trajectory analysis based on target trajectory data,
[0011] A security risk analysis is performed based on the alarm area where the spatial position of the target trajectory in the trajectory analysis result is located, wherein the alarm area is pre-configured according to the security risk level.
[0012] Preferably, the image data further includes a monocular RGB image frame, and the monocular image frame and the binocular image frame form a trinocular image frame;
[0013] Preferably, the performing target detection with a human body as a target based on any monocular image frame in the image data includes:
[0014] The trained neural network model is used to perform target detection on monocular image frames, and the intermediate false detection targets obtained during the reasoning process of the trained neural network model are corrected and eliminated online using the false detection target template used to characterize the same false detection target features.
[0015] Preferably, the neural network model comprises:
[0016] A feature extraction network for extracting features, a candidate region network for obtaining candidate box positions and candidate box classification, and a first feature of interest network for feature alignment are sequentially connected, and
[0017] A classification prediction network for predicting the classification of a target frame and a regression prediction network for predicting the position of a target frame, respectively connected to the first feature of interest network, wherein the classification prediction network also inputs a misdetected target template;
[0018] Preferably, the neural network model is trained in the following manner:
[0019] The sample target image frame and the sample misdetected image frame are input into the feature extraction network in parallel.
[0020] Based on the second image feature output by the feature extraction network and used to characterize the sample misdetected image feature, a misdetected target template is obtained.
[0021] The false detection target template is matched with the sample candidate target features output by the candidate region network, and the matching result is input into the classification prediction network, so that the classification prediction network corrects the classification confidence of the sample target box according to the matching result.
[0022] Calculate the total loss function value, which is the sum of the target box classification loss function value of the classification prediction network, the target box regression loss function value of the regression prediction network, the candidate box position loss function value of the candidate area network, and the candidate box target category loss function value. The target box classification loss function value is calculated based on the corrected sample target box classification confidence.
[0023] Freeze the model parameters of the feature extraction network, and adjust the model parameters of the candidate region network and the number of retained candidate boxes, as well as the model parameters of the classification prediction network according to the total loss function value.
[0024] Preferably, the second image feature outputted by the feature extraction network and used to characterize the sample misdetected image feature, and obtaining the misdetected target template, comprises:
[0025] Using the second alignment network of interest for feature alignment, the second image features are aligned based on the labeled information in the sample misdetected image frame to obtain the aligned second image features.
[0026] The aligned second image features are processed by the residual network and the fully connected layer in turn to obtain feature vectors of each category, where the number of feature vectors of each category is: the number of feature vectors of similar falsely detected targets output by the residual network and output by the fully connected layer,
[0027] The feature vectors of each category are fused into a synthetic feature vector as a false detection target template, and stored in a false detection database.
[0028] Preferably, the method of using the trained neural network model to perform target detection on a monocular image frame, and using a false detection target template for characterizing the same false detection target features to perform online correction and elimination of intermediate false detection targets obtained during the reasoning process of the trained neural network model, comprises:
[0029] Input the monocular image frame into the trained neural network model,
[0030] Match the candidate target feature vector output by the first alignment network of interest in the trained neural network model with the synthetic feature vector corresponding to the false detection target template.
[0031] The matching result is input into the classification prediction network in the trained neural network model, so that the classification confidence of the target frame is corrected, and the corrected classification confidence of the target frame is obtained.
[0032] The classification prediction network selects the best target box and outputs it based on the corrected target box classification confidence.
[0033] Preferably, the matching is performed in the following manner:
[0034] In the training phase, the similarity between the synthetic feature vector corresponding to the false detection target template and the sample candidate target feature vector is calculated.
[0035] In the inference stage, the similarity between the synthetic feature vector corresponding to the input misdetected target template and the candidate target feature vector is calculated.
[0036] The correction is performed as follows:
[0037] In the training stage, the current similarity is weighted to obtain the weighted similarity, the weighted similarity is inverted, and the result of the inversion operation is multiplied by the current classification confidence of the sample target frame, and the multiplication result is used to update the current classification confidence of the sample target frame to obtain the corrected classification confidence of the sample target frame;
[0038] In the inference stage, the current similarity is weighted to obtain the weighted similarity, the weighted similarity is inverted, and the result of the inversion operation is multiplied by the current classification confidence of the target box, and the multiplication result is used to update the current classification confidence of the target box to obtain the corrected classification confidence of the target box.
[0039] Preferably, the image segmentation is performed with the whole human body and / or a part of the human body as a target to obtain the segmented target image, including:
[0040] Performing dense human posture segmentation on the target to obtain a pixel region of a local human body, wherein the local human body includes at least one of a head, a torso, an upper limb, and a lower limb;
[0041] Preferably, the method of using the disparity estimation between binocular image frames in the image data to perform a target point cloud principal component analysis on the segmented target image to obtain a target disparity map includes:
[0042] Perform feature matching on the binocular images in the binocular image frame to obtain a disparity map used to characterize the disparity value of each pixel point.
[0043] The background pixels other than the target in the disparity map are removed and filtered to obtain the target disparity map.
[0044] Preferably, the performing trajectory analysis based on the target trajectory data includes:
[0045] Get the target trajectory data of the same target under different cameras, and merge the target trajectories of the same target under different cameras.
[0046] In the case where there is a missing in the merged target trajectory, the trajectory of the same target is completed to obtain the completed target trajectory data of the same target.
[0047] Based on the completed target trajectory data, trajectory prediction is performed on the target trajectory that is not tracked in the current frame and / or the target trajectory after a set time length to obtain complete target trajectory data.
[0048] Preferably, the alarm area is configured as follows:
[0049] Using the camera's internal and external parameters, the spatial position information of the pixels in the image data is obtained.
[0050] Extract the projection position information of the spatial position information of the pixel point to obtain the overhead projection position information of the working space.
[0051] In response to the user's configuration operation based on the overhead projection map, each alarm area is configured,
[0052] in,
[0053] The alarm area includes: at least one of: a first alarm area for representing a high risk, a second alarm area for representing a potential risk, and a third alarm area for representing a low risk;
[0054] Preferably, the security risk analysis includes:
[0055] When the spatial position of the target trajectory and / or the spatial position of a local human body is located in the first alarm area, it is determined to be a high risk, and a first alarm message for indicating the high risk is output.
[0056] When the spatial position of the target trajectory and / or the spatial position of a local human body is located in the second alarm area, it is determined to be a potential risk, and a second alarm information for characterizing the potential risk is output.
[0057] When the spatial position of the target trajectory and / or the spatial position of the local human body is located in the third warning area, it is determined to be a low risk, and the third warning information for indicating the low risk is output.
[0058] Preferably, the working space is a robot arm working space.
[0059] Preferably, the first alarm area is a circular projection area with the robot arm as the center and the maximum arm span radius, the second alarm area is the area within the boundary of the robot arm operation safety area, and the third alarm area is the area at the entrance of the robot arm operation space.
[0060] Preferably, the first alarm information and the second alarm information are used to control the operation of the robot arm, and the third alarm information is used to perform reliability detection on the detection result from the optical sensor located at the entrance of the robot arm's working space.
[0061] A second aspect of the present application provides a protective device for a working space, the device comprising:
[0062] The target detection module is used to perform target detection with the human body as the target based on any monocular image frame in the image data, and to track the detected target to obtain target trajectory data, and to perform image segmentation with the entire human body and / or a part of the human body as the target to obtain the segmented target image.
[0063] The target positioning module is used to use the disparity estimation between the binocular image frames in the image data to perform target point cloud principal component analysis on the segmented target image to obtain a target disparity map, obtain the spatial position information of the target based on the target disparity map, and determine the spatial position of the target trajectory in the target trajectory data according to the target spatial position information.
[0064] Trajectory analysis module, used to perform trajectory analysis based on target trajectory data,
[0065] The safety risk analysis module is used to perform safety risk analysis according to the alarm area where the spatial position of the target trajectory in the trajectory analysis result is located, wherein the alarm area is pre-configured according to the safety risk level.
[0066] An embodiment of the present application provides a method for protecting work safety, which detects and tracks a target through any monocular image frame in a multi-eye image frame, locates the entire human body and / or a part of the human body through a binocular image frame in the multi-eye image frame, and performs trajectory analysis through trajectory data, thereby improving safety assurance in the work space. Compared with a work safety protection method based on two-dimensional images, the accuracy of positioning is improved, false detections and missed detections are reduced, and the safety of local parts of the human body is also guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 A flowchart of a method for protecting operational safety provided in an embodiment of the present application.
[0068] Figure 2 A schematic diagram of the alarm area configuration in this embodiment.
[0069] Figure 3 A schematic flow chart of the robot arm operation safety protection method of this embodiment.
[0070] Figure 4 A schematic diagram of the first neural network model for object detection.
[0071] Figure 5 A schematic diagram of a second neural network model for removing falsely detected targets during trained target detection.
[0072] Figure 6 A schematic diagram of training the second neural network model.
[0073] Figure 7 It is a schematic diagram of the segmentation of various local parts of the human body in this embodiment.
[0074] Figure 8 It is a schematic diagram of a disparity map obtained during the principal component analysis of a target point cloud with a human body as the target in this embodiment, a disparity map after removing background pixels, and a filtered target disparity map.
[0075] Fig. 9 A schematic diagram of calculating the three-dimensional spatial coordinates of the target point cloud based on the binocular image frame.
[0076] Fig.10 A schematic diagram of the trajectory analysis and safety risk analysis subject of this embodiment.
[0077] Fig.11A schematic diagram for comparing target detection based on two-dimensional images and target detection based on three-dimensional images.
[0078] Fig.12 A schematic diagram of a protective device for a working space according to an embodiment of the present application.
[0079] Fig.13 Another schematic diagram of the protective device for the working space of an embodiment of the present application. DETAILED DESCRIPTION
[0080] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.
[0081] An embodiment of the present application provides a method for protecting work safety, which improves safety assurance in the work space by performing intelligent analysis on image data in the work space.
[0082] See also Figure 1 As shown, Figure 1 A flowchart of a method for protecting operation safety provided in an embodiment of the present application. The method includes:
[0083] Step 101, acquiring image data in the workspace, wherein the image data includes a binocular image frame,
[0084] As an example, the environmental image of the working space can be collected by a binocular camera, or by a trinocular camera. In this way, the image data can also include a monocular image frame. Preferably, the monocular image frame is an RGB image.
[0085] Step 102, based on any monocular image frame in the image data, target detection is performed with the human body as the target, and the detected target is tracked to obtain target trajectory data, and image segmentation is performed with the entire human body and / or a part of the human body as the target to obtain a segmented target image,
[0086] As an example, the monocular image frame may be one of the binocular image frames, or may be an independent monocular image frame.
[0087] As an example, object detection is performed as follows:
[0088] The trained neural network model is used to perform target detection on monocular image frames, and the false detection target template used to characterize the same false detection target features is used to perform online correction and elimination on the intermediate false detection targets obtained during the reasoning process of the trained neural network model, wherein the false detection target template is equivalent to the registered template stored in the false detection database.
[0089] As an example, a neural network model includes:
[0090] A feature extraction network for extracting features, a candidate region network for obtaining candidate box positions and candidate box classification, and a first feature of interest network for feature alignment, which are connected in sequence, and
[0091] A classification prediction network for predicting the classification of a target frame and a regression prediction network for predicting the position of a target frame, respectively connected to the first feature of interest network, wherein the classification prediction network also inputs a misdetected target template;
[0092] In the inference phase:
[0093] Input the monocular image frame into the trained neural network model,
[0094] Match the candidate target feature vector output by the first alignment network of interest in the trained neural network model with the synthetic feature vector corresponding to the false detection target template.
[0095] The matching result is input into the classification prediction network in the trained neural network model, so that the classification confidence of the target frame is corrected, and the corrected classification confidence of the target frame is obtained.
[0096] The classification prediction network selects the best target box and outputs it based on the corrected target box classification confidence.
[0097] As an example, a neural network model is trained as follows:
[0098] The sample target image frame and the sample misdetected image frame are input into the feature extraction network in parallel.
[0099] Based on the second image feature output by the feature extraction network and used to characterize the sample misdetected image feature, a misdetected target template is obtained.
[0100] The false detection target template is matched with the sample candidate target features output by the candidate region network, and the matching result is input into the classification prediction network, so that the classification prediction network corrects the classification confidence of the sample target box according to the matching result.
[0101] Calculate the total loss function value, which is the sum of the target box classification loss function value of the classification prediction network, the target box regression loss function value of the regression prediction network, the candidate box position loss function value of the candidate area network, and the candidate box target category loss function value. The target box classification loss function value is calculated based on the corrected sample target box classification confidence.
[0102] Freeze the model parameters of the feature extraction network, and adjust the model parameters of the candidate region network and the number of retained candidate boxes, as well as the model parameters of the classification prediction network according to the total loss function value.
[0103] As an example, the false detection target template is obtained as follows:
[0104] Using the second alignment network of interest for feature alignment, the second image features are aligned based on the labeled information in the sample misdetected image frame to obtain the aligned second image features.
[0105] The aligned second image features are processed by the residual network and the fully connected layer in turn to obtain the feature vectors of each category.
[0106] The feature vectors of each category are fused into a synthetic feature vector as a false detection target template, and stored in the false detection database as a registration template.
[0107] As an example, image segmentation is performed as follows:
[0108] The target is subjected to dense human posture segmentation to obtain a pixel region of a local human body, wherein the local human body includes at least one of a head, a torso, an upper limb, and a lower limb.
[0109] Step 103, using the disparity estimation between binocular image frames in the image data, a principal component analysis of the target point cloud is performed on the segmented target image to obtain a target disparity map.
[0110] As an example, feature matching is performed on the binocular images in the binocular image frame to obtain a disparity map for representing the disparity value of each pixel point, and background pixels other than the target in the disparity map are removed and filtered to obtain a target disparity map.
[0111] Step 104, based on the target disparity map, obtain the spatial position information of the target, and determine the spatial position of the target trajectory in the target trajectory data according to the spatial position information of the target.
[0112] As an example, the three-dimensional spatial position coordinates of each target pixel point in the target disparity map are calculated using the intrinsic and extrinsic parameters of the camera to improve the accuracy of target positioning.
[0113] Step 105, performing trajectory analysis based on the target trajectory data,
[0114] As an example, target trajectory data of the same target under different cameras is obtained, and the target trajectories of the same target under different cameras are merged. If there is a missing part in the merged target trajectory, the trajectory of the same target is completed to obtain the completed target trajectory data of the same target. Based on the completed target trajectory data, the target trajectory that is not tracked in the current frame and / or the target trajectory after a set time length is predicted to obtain the complete target trajectory data.
[0115] Step 106 , outputting alarm information corresponding to the alarm area according to the alarm area where the spatial position of the target trajectory in the trajectory analysis result is located, wherein the alarm area is pre-configured according to the security risk level.
[0116] As an example, when the spatial position of the target trajectory enters the first alarm area, the first alarm information for indicating a high risk is output.
[0117] When the spatial position of the target trajectory enters the second alarm area, outputting the second alarm information for characterizing the potential risk,
[0118] When the spatial position of the target trajectory enters the third warning area, a third warning message for indicating a low risk is output.
[0119] The operation safety protection method provided in the embodiment of the present application obtains the positioning information and trajectory data of the target through binocular images, and makes corresponding alarms through the alarm areas of different safety risk levels where the trajectory data is located. Compared with the operation safety protection method based on two-dimensional images, the positioning accuracy is improved, the false detection and missed detection are reduced, and the false detection rate of the target is reduced through online false detection correction and elimination.
[0120] To facilitate the understanding of the present application, the following explanation is given by taking the protection of robot arm operation safety as an example. It should be understood that the present application is not limited to robot arm operations, but is also applicable to the protection of safety of other operations.
[0121] Machinery safety implementation includes:
[0122] Protection methods, such as small working gaps, equipment installation locations, etc.;
[0123] Guarding devices such as safety fences, locking sliding doors, etc.;
[0124] Protection devices, such as interlocking devices, enabling devices, hold-to-run devices, two-hand operation devices, sensitive protection, active photoelectric protection;
[0125] Supplementary protective devices such as emergency stop buttons, lockout / tagout, access facilities, shutdown performance monitoring, safety pins, etc.
[0126] The applicant has found that both common protective devices and supplementary protective devices have potential safety hazards and unsolvable problems to varying degrees.
[0127] For example, sensitive protection generally uses pressure-sensitive mats and pressure-sensitive strips to sense whether there are people standing in the laid area. However, it is difficult for pressure-sensitive mats to cover all blind spots in the working area, and it cannot solve the problem of people climbing up with their feet off the ground when repairing equipment. It is also impossible to distinguish whether the pressure-sensitive trigger area is a person or a pile of goods / AGV entering.
[0128] For example, in order to facilitate manual / AGV loading and unloading, some fences will use light barriers to detect entry and exit events instead of physical sliding doors. However, light barriers may have the risk of false detection, including: when the light barrier's beam density is insufficient, there will be missed reports of physical entry, it cannot distinguish whether the trigger object is a person or an AGV, it can only detect whether someone passes through but not whether they enter or leave, it cannot detect whether the person is still present, and illegal entry from non-entrances cannot be detected, etc.
[0129] For another example, when using general visual sensors to detect and analyze people, due to the perspective relationship in the image, it is impossible to accurately locate the position of the people, and there will be problems such as people inside the fence being mistakenly detected as being outside the fence, or people outside the fence being mistakenly detected as being inside the fence. Since existing target detection algorithms have a certain probability of missed detection and false detection, missed detection will cause safety risks, and false detection will cause erroneous shutdowns, thereby reducing production efficiency. In addition, in some working environments, the safety space is small, and there is a situation where local parts such as hands, feet, and head enter the danger zone first. Only target detection and positioning of the human body is not enough to identify such dangers.
[0130] In summary, the defects of existing protection methods can be briefly summarized as shown in Table 1.
[0131]
[0132] In view of this, an embodiment of the present application provides a method for protecting operation safety to improve operation safety.
[0133] As an example, in the working space of the robot arm, an image acquisition device for capturing images is deployed. For example, a binocular camera is deployed above the fence in the working space of the robot arm. A trio of cameras can also be deployed, where two cameras are used to capture binocular images and the other camera is used to capture RGB images. To avoid blind spots in monitoring, binocular or trio of cameras can also be deployed at locations where blind spots exist.
[0134] In this embodiment, an alarm area is pre-configured for the robot arm operation space to be protected. Figure 2 As shown, Figure 2A schematic diagram of the configuration of the alarm area of this embodiment, the stereogram in the figure is an example of an image screen captured by the image acquisition device. Based on the pixel position information in the captured image, the spatial position information of the pixel in the captured image can be obtained by using the internal and external parameters of the camera, and the projection position information can be extracted through the spatial position information of the pixel. For example, ignoring the height information in the spatial position information, the overhead projection position information of the robot arm working space can be obtained, and these projection position information form a top-down projection map; the user can configure each alarm area based on the top-down projection map, so that the configured alarm area corresponds to its spatial position. As an example, each alarm area is distinguished according to the level of safety risk. For example, the circular area with the robot arm coordinates as the center and the maximum arm span as the radius is the first alarm area for characterizing high-risk risks, such as the red marked area in the figure, the area within the boundary of the robot arm operation safety area is the second alarm area for characterizing potential risks, such as the yellow marked area in the figure, and the area at the entrance of the robot arm working space is the third area for characterizing low-risk risks, such as the blue marked area in the figure. The configuration method may be in response to configuration operations such as editing operations and input operations performed by the user in the projection coordinate system of the robot arm working area through the human-computer interaction interface.
[0135] See also Figure 3 As shown, Figure 3 This is a flow chart of a method for protecting the safety of robot arm operation in this embodiment. The method includes:
[0136] Step 301, obtaining image data in the robot arm working space,
[0137] As an example, the image data includes a binocular image frame from a binocular camera and an RBG image frame from an RGB camera.
[0138] Step 302: Based on the image data, target detection is performed with the human body as the target.
[0139] As an example, target detection is performed based on the left or right eye image, or the RGB image in the binocular camera. The target detection method can be any deep learning-based target detection method, and a target frame with the human body as the target is output to obtain a target detection result. If there are multiple human body detection results, a target queue containing multiple human body target frame information is output.
[0140] See also Figure 4 As shown, Figure 4A schematic diagram of a first neural network model for target detection. The first neural network model in the figure includes a feature extraction network such as a backbone network for feature extraction, a candidate region network (RPN) for obtaining candidate box positions and candidate box classifications, a first alignment network of interest (RIO align) for feature alignment, a classification prediction network such as a classification prediction head cls for predicting target box classification, and a regression prediction network such as a regression prediction head reg for predicting target box positions.
[0141] Experiments have shown that the misdetected target box is not due to insufficient network regression accuracy, but to incorrect target box classification. Based on this, when training the neural network model, on the basis of freezing the feature extraction network, that is, keeping the parameters of the feature extraction network unchanged temporarily, only updating the RPN classification parameters and the classification prediction head cls classification parameters, and adjusting the number of candidate boxes retained by the RPN. The classification prediction head cls outputs the target category of the target box, and the regression prediction head reg outputs the location information of the target box. The loss function of the model consists of four parts: the candidate box position loss function value of the RPN module, the candidate box target category loss function value, the target box regression loss function value of the regression prediction head, and the target classification loss function value of the classification prediction head. The total loss function value is expressed mathematically as follows:
[0142] in, L total is the total loss function value, L fpn_local is the candidate box position loss function value, L fpn_cls is the target category loss function value of the candidate box, L cls is the target classification loss function value of the classification prediction head, L reg is the target box regression loss function value of the regression prediction head; target box regression loss function value L reg Using Smooth L1 loss function, target classification loss function value L cls Use binary cross entropy loss function.
[0143] Since false detection may cause the robot to stop working, the false detection target is corrected and removed from the target detection results. Figure 5 As shown, Figure 5 A schematic diagram of a second neural network model for removing falsely detected targets in a trained target detection process, wherein falsely detected target templates from a falsely detected database are input into a feature matching network to perform feature matching between the falsely detected target templates and candidate target features in a candidate frame, wherein the candidate target features are intermediate falsely detected target features.
[0144] As an example, assume that the feature vector of the i-th misdetected target template is x i , the feature vector of the candidate target is p i , calculate the cosine similarity between the feature vector of the false detection target template and the feature vector of the candidate target, and the mathematical formula is as follows:
[0145]
[0146] in, s i cos is the cosine similarity, ‖‖ represents the modulus of the vector.
[0147] Assume that the current classification confidence of any target box output by the classification prediction head is s i , the current target box classification confidence is corrected according to the weighted coefficient and cosine similarity. The corrected mathematical expression is as follows:
[0148]
[0149] Among them, α is the weighting coefficient, s is the corrected target box classification confidence, 1-α s i cos For α s i cos The negation operation.
[0150] The classification prediction head corrects the classification confidence of each target frame, and selects the target frame with the best confidence as the best target frame according to the corrected target frame classification confidence, thereby suppressing and eliminating the false positive frames. Since the suppression and elimination of false positive frames are carried out simultaneously with target detection, Figure 5 The second neural network model in can be called a neural network model with online correction and elimination capabilities. During the correction process, the target detection box regression calculation is not affected, and only the misdetected targets are suppressed and removed in the classification branch, realizing the screening and removal of few-shot misdetected targets.
[0151] The second neural network model can be trained as follows. Figure 6 As shown, Figure 6A schematic diagram for training the second neural network model. The figure includes a feature extraction network, a first alignment network of interest (RIO align), a second alignment network of interest, a candidate region network (RPN), a residual network, a fully connected layer, a classification prediction head cls, and a regression prediction head reg, and a feature matching network. Among them, the feature extraction network may include a backbone network and a feature pyramid network; the sequentially connected feature extraction network, the candidate region network, the first alignment network of interest, the classification prediction head cls, and the regression prediction head connected to the first alignment network of interest are used for target detection, the sequentially connected feature extraction network, the second alignment network of interest, the residual network, and the fully connected layer are used to obtain the misdetected target template, and the feature matching network is used to match the misdetected target template output by the fully connected layer with the sample features output by the first alignment network of interest, so as to correct the classification confidence of the target box output by the classification prediction head according to the sample matching result.
[0152] Specifically, the sample target image frame and the sample false detection image frame are respectively input into the feature extraction network to obtain the first sample image feature for characterizing the sample target image frame information and the second sample image feature for characterizing the sample false detection image frame information. The RIO information marked in the sample false detection image frame and the second sample image feature are input into the second alignment network of interest. The second alignment network of interest outputs the aligned second sample image feature vector. The second sample image feature is sequentially passed through the residual network and the fully connected layer to obtain the feature vectors of various false detection targets. In the feature metric space, there is a difference between the direction of the feature vector of the false detection target and the direction of the feature vector of the positive detection target, while the directions of the feature vectors of the same false detection target are relatively close. In this way, the feature vectors of various types of sample false detection targets are fused into a synthetic feature vector as the false detection target template of the various types of false detection targets, and stored in the false detection database. For example, Figure 6 The fully connected layer in the middle outputs k types of false positive target feature vectors, and the size of each feature vector is 1×2048.
[0153] Assume that any falsely detected target feature vector is y j , then the synthetic feature vector is the average value of the feature vectors of various types of false detection targets output by the fully connected layer, which can be expressed mathematically as follows:
[0154]
[0155] Among them, k is the number of false detection target feature vectors output by the fully connected layer of similar false detection targets, that is, k similar false detection targets, and x is the synthetic feature vector, that is, the feature vector of the false detection target template.
[0156] Calculate the sample similarity between the feature vector of the misdetected target template and the aligned second sample image feature. According to the sample similarity and the corrected mathematical expression, correct the confidence of the sample target box classification output by the classification prediction head. Then, according to the total loss function value accumulated by the candidate box position loss function value of the RPN module, the candidate box target category loss function value, the target box regression loss function value of the regression prediction head, and the target classification loss function value of the classification prediction head, adjust the RPN classification parameters, the classification prediction head cls classification parameters, and the number of candidate boxes retained by the RPN. The target classification loss function value is calculated based on the corrected confidence of the sample target box classification.
[0157] The false detection database can also pre-store false detection targets and their features. The false detection targets and their features come from two sources:
[0158] One source is that when manually resetting the robot arm, the detected detection result is manually confirmed to be a false detection target. The manual confirmation can be operated on the computer screen or by operating the forced reset button;
[0159] Another source is the target detection results automatically obtained when there is no grating trigger information during the movement of the robot arm and the starting point of the target trajectory generation is not at the fence entrance.
[0160] Among them, the features of the misdetected targets are used to extract feature vectors using the feature extraction network used in the target detection process.
[0161] Step 303, based on the target detection result, target tracking is performed to obtain the movement trajectory information of each target, wherein each trajectory information contains the target frame information of each historical moment, that is, each image frame.
[0162] Step 304, based on any target image data from the binocular camera, image segmentation is performed with the whole human body and / or part of the human body as the target,
[0163] As an example, based on the left eye image in the binocular camera, the dense human posture segmentation algorithm is used to segment the image to segment the pixel areas of local parts such as the head, hands, and feet in the image coordinate system and represent them in the form of masks, see Figure 7 As shown, Figure 7 This is a schematic diagram of the segmentation of local parts of the human body in this embodiment. The segmented local parts include five areas: head, left hand, right hand, left foot, and right foot.
[0164] Step 305 , using the disparity estimation between binocular image frames in the image data, a principal component analysis of the target point cloud is performed on the segmented target image to obtain a target disparity map.
[0165] Although target detection and posture segmentation algorithms have very good effects on target processing, in some complex situations or scenes that are very different from the training set, there will still be problems with target frame offset or segmentation errors. In addition, due to the lack of rich texture of the environment or target image, a certain number of erroneous points, i.e., flying points, will appear in the target point cloud computing, which will affect the accuracy of target positioning. For the point cloud in the target area that has interference points from other backgrounds or targets due to inaccurate target detection or image segmentation, the principal component analysis of the target point cloud can filter the point cloud, eliminate the background or other target point sets and flying points, and use the correct three-dimensional point set in the target area as the main component to achieve accurate positioning of the target.
[0166] In this embodiment, the principal component analysis of the target point cloud with the human body as the target includes:
[0167] The first step is to model the disparity map to remove background pixels other than the target in the disparity map, and obtain the disparity map after removing the background pixels, that is, the target disparity map.
[0168] The second step is to filter the disparity map after removing the background pixels to obtain a filtered target disparity map.
[0169] See also Figure 8 As shown, Figure 8 It is a schematic diagram of a disparity map obtained during the principal component analysis of a target point cloud with a human body as the target in this embodiment, a disparity map after removing background pixels, and a filtered target disparity map.
[0170] As an example, the disparity map modeling process is as follows:
[0171] The disparity matching algorithm is used to match the pixels of the left and right image frames in the binocular image frame to obtain the disparity value of each pixel of the binocular image frame; the disparity value is matched according to the position of each pixel to obtain the disparity map of the binocular image frame.
[0172] For any pixel, remove the disparity map frame with abnormal disparity value of the pixel in the disparity map of the adjacent N frames, calculate the mean or variance of the disparity value of the pixel in the remaining disparity map frames, and obtain the background disparity value of the pixel. For example, suppose the disparity value of the i-th pixel in the j-th frame disparity map is u ij , then the background disparity value of the pixel v ij The mathematical expression is as follows:
[0173]
[0174] Wherein, M is the total number of frames of the disparity map where the abnormal disparity value of the pixel i removed from the disparity map of adjacent N frames is located, and is the remaining number of frames of the disparity map.
[0175] For any frame disparity map, the disparity value of each pixel in the target area of the disparity map is compared with the background disparity value of the pixel. If the difference result is less than the preset threshold Thr, the pixel is considered to be a background pixel, and the background pixel is removed from the disparity map to obtain a disparity map without the background pixel. The difference result can be a difference value or a ratio value.
[0176] As an example, the process of filtering based on the disparity map after removing background pixels is as follows:
[0177] (1) Based on the disparity map after removing background pixels, calculate the distance value from each pixel to the camera imaging plane.
[0178] (2) Sort the calculated distance values in ascending order.
[0179] (3) Approximate Gaussian filtering is performed on the sorted distance values. The calculation formula is as follows:
[0180]
[0181] in, dist_filter i is the approximate Gaussian filtering result of the sorted i-th distance value, dist i-1 is the i-1th distance value, dist i-2 is the i-2th distance value, dist i+1 is the i+1th distance value, dist i+2 is the i+2th distance value.
[0182] (4) Among the sorted distance values, the first-order derivative of the distance value is calculated according to the approximate Gaussian filtering results of two adjacent distance values. The calculation formula is as follows:
[0183]
[0184] in, dist_d i is the i-th distance value dist i The first derivative of dist_filter i-1 is the i-1th distance value dist i-1 The approximate Gaussian filtering result is:
[0185] (5) Sort the first-order derivatives of each distance value in ascending order to obtain a first-order derivative sequence.
[0186] (6) Calculate the mean of all first-order derivatives in the first-order derivative sequence, obtain the first-order derivatives that are less than the mean from the first-order derivative sequence, calculate the mean or variance of the obtained first-order derivatives, use the calculated mean or variance as the derivative threshold distThr, and mark the distance value of the first-order derivative greater than the derivative threshold distThr as the extreme point.
[0187] (7) Suppress the maximum distance value of all extreme points.
[0188] (8) Sort the pixel position indexes corresponding to the unsuppressed maximum distance values, and calculate the number of pixels between the sorted adjacent pixel position indexes. The area with the largest number of pixels is the final target pixel set to obtain the filtered target disparity map.
[0189] Step 306, according to the pre-calibrated camera external parameters, that is, the camera's three-dimensional spatial position and the three angles of pitch, tilt, and roll, calculate the depth value corresponding to each target pixel in the filtered target disparity map, and then calculate the three-dimensional spatial coordinates of each target pixel to obtain the three-dimensional spatial information of the target point cloud as the target three-dimensional point coordinate set;
[0190] See also Fig. 9 As shown, Fig. 9 A schematic diagram of calculating the three-dimensional spatial coordinates of the target point cloud based on the binocular image frame. In the figure, the color image is a schematic diagram of the depth map. Step 307, based on the three-dimensional spatial coordinates of each target point cloud, ignoring the height coordinate value, the position information of each target point cloud in the top view can be obtained, and the position information is used as the target track position of the target frame, for example, the head position is the target track position, so as to compare with the configured alarm area.
[0191] As an example, the mean value of the three-dimensional point coordinates of all target point clouds in the target three-dimensional point coordinate set is calculated as the target positioning result.
[0192] As another example, the mean value of the three-dimensional point coordinates of a local area, such as a head area, in the target three-dimensional point coordinate set is calculated to obtain the head position as the target trajectory position.
[0193] Step 308, obtaining the target trajectory under different cameras,
[0194] If there are multiple cameras working together, the trajectories of the same target under different cameras are obtained, the trajectories of the same target under multiple cameras are archived, and the archived trajectories of the same target are merged;
[0195] As an example, when trajectories are merged, the target frames contained in the trajectory data are judged in turn according to the facial features, body appearance features, and trajectory features. The facial features are identified based on the face recognition algorithm. When the facial features are consistent, they are determined to be the same target. Otherwise, the body appearance features are compared based on the body weight recognition algorithm. When they are consistent, they are determined to be the same target. Otherwise, the trajectory features between the trajectory segments with overlapping time, such as trajectory curvature, direction, etc., are compared. When they are consistent, they are determined to be the same target. Comparison refers to judging the similarity or consistency of two or more objects according to set standards or requirements.
[0196] Step 309: when the trajectory of the same target is missing in a certain time period after merging, the trajectory of the same target is completed to obtain the completed target trajectory data of the same target.
[0197] For example, using adjacent trajectory points before and after the missing time period as endpoints, interpolation generates trajectory data within the missing time period;
[0198] Step 310, based on the acquired trajectory data, trajectory prediction is performed.
[0199] As an example, the trajectory prediction may be performing trajectory prediction on the trajectory of a target that is not detected in the current frame, for example, performing trajectory prediction on a person who is located in a blind spot due to occlusion, and / or, if the latest trajectory point in the trajectory data of the same target is not in the blue entrance area, performing position prediction, for example, performing trajectory prediction based on Kalman filtering, so as to supplement the trajectory points of the current frame;
[0200] The trajectory prediction can also be the trajectory prediction of the normally tracked target in the current frame, so as to predict its trajectory data after a set time in the future, for example, predicting the trajectory data after t seconds, where t is generally a small value such as 1 to 3 seconds.
[0201] Through steps 308 to 310, complete target trajectory data of each same target can be obtained.
[0202] Step 311, outputting alarm information according to the alarm area where the target trajectory position is located.
[0203] As an example, by comparing the target trajectory position with the position of the alarm area, the alarm area where the target trajectory position is located can be determined.
[0204] When the current position or the predicted future position of the target trajectory has entered the first alarm area, and / or the position of a local human body has been located in the first alarm area, for example, a red area, a first alarm message for indicating a high risk is output;
[0205] When the current position or the predicted future position of the target trajectory has entered the second alarm area, and / or the position of a local human body has entered the second alarm area, for example, the yellow area, a second alarm message for characterizing a potential risk is output;
[0206] When the current position or future predicted position of the target trajectory has entered the third alarm area, and / or the position of a local human body has entered the third alarm area, the third alarm information used to indicate that there is a target in the second alarm area is continuously output to confirm the false alarm when the grating is triggered.
[0207] The first and second alarm information can be used to directly control the operating status of the robot arm, including whether to stop, slow down, or allow startup, etc.; the third alarm information is used when an optical sensor, such as a light grating, is blocked and triggered, to jointly determine whether it is triggered by the AGV or a person, and then decide whether to stop the operation of the robot arm, so as to perform reliability detection on the detection results from the optical sensor located at the entrance of the robot arm's working space.
[0208] See also Fig.10 As shown, Fig.10 This is a schematic diagram of the trajectory analysis and safety risk analysis subject of this embodiment. In the left figure, the target trajectory can be generated based on one or more binocular cameras, and each target trajectory is sent to an industrial computer or other computing device such as a server. The industrial computer or other computing device generates an alarm information output based on the received multi-camera trajectory data. In the right figure, the industrial computer or computing device used for trajectory analysis and safety risk analysis can be replaced by a camera.
[0209] In this embodiment, the whole human body and / or part of the human body is taken as the target, and the safety protection of the local limbs is solved. Through the target trajectory generation, merging, completion and prediction of multiple cameras, the complete trajectory generation after the person enters the working space is guaranteed, and missed detection and false detection are avoided. The false detection target template can realize the automatic removal of false detection, which is conducive to automatically avoiding false detection caused by the environment in complex and changeable application scenarios. For example, see Fig.11 As shown, Fig.11 This is a schematic diagram of the comparison between target detection based on two-dimensional images and target detection based on three-dimensional images. It can be seen from the figure that target detection based on two-dimensional images has false positives, while target detection based on three-dimensional images has no false positives.
[0210] See also Fig.12 As shown, Fig.12 This is a schematic diagram of a protective device for a working space in an embodiment of the present application. The device includes:
[0211] The target detection module is used to perform target detection with the human body as the target based on any monocular image frame in the image data, and to track the detected target to obtain target trajectory data, and to perform image segmentation with the entire human body and / or a part of the human body as the target to obtain the segmented target image.
[0212] The target positioning module is used to use the disparity estimation between the binocular image frames in the image data to perform target point cloud principal component analysis on the segmented target image to obtain a target disparity map, obtain the spatial position information of the target based on the target disparity map, and determine the spatial position of the target trajectory in the target trajectory data according to the target spatial position information.
[0213] Trajectory analysis module, used to perform trajectory analysis based on target trajectory data,
[0214] The safety risk analysis module is used to perform safety risk analysis according to the alarm area where the spatial position of the target trajectory in the trajectory analysis result is located, wherein the alarm area is pre-configured according to the safety risk level.
[0215] As an example, the device further includes:
[0216] The configuration module is used to use the camera's internal and external parameters to obtain the spatial position information of the pixel points in the image data, extract the projection position information of the spatial position information of the pixel points, obtain the overhead projection position information of the working space, and configure each alarm area in response to the user's configuration operation based on the overhead projection map.
[0217] As yet another example, the device further includes:
[0218] The image acquisition module is used to obtain image data in the working space.
[0219] See also Fig.13 As shown, Fig.13 Another schematic diagram of a protective device for work safety according to an embodiment of the present application. The device includes a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the protective method for work safety according to an embodiment of the present application. The device can be used to supplement existing protective devices. The device can also include an image collector, such as several cameras, so that the device can be used as an independent protective system.
[0220] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0221] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0222] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the method for protecting job safety described in the embodiment of the present application are implemented.
[0223] As for the apparatus / network-side device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0224] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0225] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for protecting operation safety, characterized in that: The method includes: Acquire image data in the workspace, wherein the image data includes a binocular image frame, Based on any monocular image frame in the image data, target detection is performed with the human body as the target, and the detected target is tracked to obtain target trajectory data, and image segmentation is performed with the entire human body and / or a part of the human body as the target to obtain the segmented target image. Using the disparity estimation between binocular image frames in the image data, the principal component analysis of the target point cloud is performed on the segmented target image to obtain the target disparity map. Based on the target disparity map, the spatial position information of the target is obtained, and according to the spatial position information of the target, the spatial position of the target trajectory in the target trajectory data is determined, Perform trajectory analysis based on target trajectory data, Perform security risk analysis based on the alarm area where the spatial position of the target trajectory in the trajectory analysis result is located, where the alarm area is pre-configured according to the security risk level; The method of performing target detection with a human body as the target based on any monocular image frame in the image data includes: The trained neural network model is used to detect the target in the monocular image frame, and the false detection target template used to characterize the same false detection target features is used to perform online correction and elimination of the intermediate false detection targets obtained during the reasoning process of the trained neural network model. The method of obtaining the falsely detected target template includes: When training the neural network model, the sample false detection image is input into the feature extraction network in the neural network model, and a second image feature for characterizing the feature of the sample false detection image is output. Using the second alignment network of interest for feature alignment, the second image features are aligned based on the labeled information in the sample misdetected image frame to obtain the aligned second image features. The aligned second image features are processed by the residual network and the fully connected layer in turn to obtain feature vectors of each category, where the number of feature vectors of each category is: the number of feature vectors of similar falsely detected targets output by the residual network and output by the fully connected layer, The feature vectors of each category are fused into a synthetic feature vector as a false detection target template, and stored in a false detection database.
2. The protection method according to claim 1, characterized in that: The image data also includes a monocular RGB image frame, and the monocular image frame and the binocular image frame form a trinocular image frame.
3. The protection method according to claim 2, characterized in that: The neural network model includes: A feature extraction network for extracting features, a candidate region network for obtaining candidate box positions and candidate box classification, and a first feature of interest network for feature alignment are sequentially connected, and A classification prediction network for predicting the classification of a target frame and a regression prediction network for predicting the position of a target frame, respectively connected to the first feature of interest network, wherein the classification prediction network also inputs a misdetected target template; The neural network model is trained as follows: When the sample misdetected image frame is input into the feature extraction network, the sample target image frame is input into the feature extraction network in parallel. The false detection target template is matched with the sample candidate target features output by the candidate region network, and the matching result is input into the classification prediction network, so that the classification prediction network corrects the classification confidence of the sample target box according to the matching result. Calculate the total loss function value, which is the sum of the target box classification loss function value of the classification prediction network, the target box regression loss function value of the regression prediction network, the candidate box position loss function value of the candidate area network, and the candidate box target category loss function value. The target box classification loss function value is calculated based on the corrected sample target box classification confidence. Freeze the model parameters of the feature extraction network, and adjust the model parameters of the candidate region network and the number of retained candidate boxes, as well as the model parameters of the classification prediction network according to the total loss function value.
4. The protection method according to claim 3, characterized in that: The method uses the trained neural network model to detect the target on the monocular image frame, and uses the false detection target template used to characterize the same false detection target features to perform online correction and elimination of the intermediate false detection targets obtained in the reasoning process of the trained neural network model, including: Input the monocular image frame into the trained neural network model, The candidate target feature vector output by the first alignment network of interest in the trained neural network model is matched with the synthetic feature vector corresponding to the false detection target template, where the candidate target is the intermediate false detection target. The matching result is input into the classification prediction network in the trained neural network model, so that the classification confidence of the target frame is corrected, and the corrected classification confidence of the target frame is obtained. The classification prediction network selects the best target box and outputs it based on the corrected target box classification confidence.
5. The protection method according to claim 4, characterized in that: The matching is performed as follows: In the training phase, the similarity between the synthetic feature vector corresponding to the false detection target template and the sample candidate target feature vector is calculated. In the inference stage, the similarity between the synthetic feature vector corresponding to the input misdetected target template and the candidate target feature vector is calculated. The correction is performed as follows: In the training stage, the current similarity is weighted to obtain the weighted similarity, the weighted similarity is inverted, and the result of the inversion operation is multiplied by the current classification confidence of the sample target frame, and the multiplication result is used to update the current classification confidence of the sample target frame to obtain the corrected classification confidence of the sample target frame; In the inference stage, the current similarity is weighted to obtain the weighted similarity, the weighted similarity is inverted, and the result of the inversion operation is multiplied by the current classification confidence of the target box, and the multiplication result is used to update the current classification confidence of the target box to obtain the corrected classification confidence of the target box.
6. The protection method according to claim 1, characterized in that: The step of performing image segmentation on the whole human body and / or a part of the human body to obtain the segmented target image includes: Performing dense human posture segmentation on the target to obtain a pixel region of a local human body, wherein the local human body includes at least one of a head, a torso, an upper limb, and a lower limb; The method utilizes the disparity estimation between binocular image frames in the image data to perform a target point cloud principal component analysis on the segmented target image to obtain a target disparity map, including: Perform feature matching on the binocular images in the binocular image frame to obtain a disparity map used to characterize the disparity value of each pixel point. The background pixels other than the target in the disparity map are removed and filtered to obtain the target disparity map.
7. The protection method according to claim 1, characterized in that: The performing trajectory analysis based on the target trajectory data includes: Get the target trajectory data of the same target under different cameras, and merge the target trajectories of the same target under different cameras. In the case where there is a missing in the merged target trajectory, the trajectory of the same target is completed to obtain the completed target trajectory data of the same target. Based on the completed target trajectory data, trajectory prediction is performed on the target trajectory that is not tracked in the current frame and / or the target trajectory after a set time length to obtain complete target trajectory data.
8. The protection method according to claim 1, characterized in that: The alarm areas are configured as follows: Using the camera's internal and external parameters, the spatial position information of the pixels in the image data is obtained. Extract the projection position information of the spatial position information of the pixel point to obtain the overhead projection position information of the working space. In response to the user's configuration operation based on the overhead projection map, each alarm area is configured, in, The alarm area includes: at least one of: a first alarm area for representing a high risk, a second alarm area for representing a potential risk, and a third alarm area for representing a low risk; The security risk analysis includes: When the spatial position of the target trajectory and / or the spatial position of a local human body is located in the first alarm area, it is determined to be a high risk, and a first alarm message for indicating the high risk is output. When the spatial position of the target trajectory and / or the spatial position of a local human body is located in the second alarm area, it is determined to be a potential risk, and a second alarm information for characterizing the potential risk is output. When the spatial position of the target trajectory and / or the spatial position of the local human body is located in the third warning area, it is determined to be a low risk, and the third warning information for indicating the low risk is output.
9. The protection method according to claim 8, characterized in that: The working space is a robot arm working space, The first alarm area is a circular projection area with the robot arm as the center and the maximum arm span radius, the second alarm area is the area within the boundary of the robot arm's operating safety area, and the third alarm area is the area at the entrance of the robot arm's operating space. The first alarm information and the second alarm information are used to control the operation of the robot arm, and the third alarm information is used to perform reliability detection on the detection result from the optical sensor located at the entrance of the robot arm's working space.
10. A protective device for a working space, characterized in that: The device includes: The target detection module is used to perform target detection with the human body as the target based on any monocular image frame in the image data, and to track the detected target to obtain target trajectory data, and to perform image segmentation with the entire human body and / or a part of the human body as the target to obtain the segmented target image. The target positioning module is used to use the disparity estimation between the binocular image frames in the image data to perform target point cloud principal component analysis on the segmented target image to obtain a target disparity map, obtain the spatial position information of the target based on the target disparity map, and determine the spatial position of the target trajectory in the target trajectory data according to the target spatial position information. Trajectory analysis module, used to perform trajectory analysis based on target trajectory data, A security risk analysis module, used to perform security risk analysis based on the alarm area where the spatial position of the target trajectory in the trajectory analysis result is located, wherein the alarm area is pre-configured according to the security risk level; Wherein, in the target detection module, the target detection with the human body as the target is performed based on any monocular image frame in the image data, including: The trained neural network model is used to detect the target on the monocular image frame, and the false detection target template used to characterize the same false detection target features is used to perform online correction and elimination on the intermediate false detection targets obtained during the reasoning process of the trained neural network model; the method of obtaining the false detection target template includes: When training the neural network model, the sample false detection image is input into the feature extraction network in the neural network model, and a second image feature for characterizing the feature of the sample false detection image is output. Using the second alignment network of interest for feature alignment, the second image features are aligned based on the labeled information in the sample misdetected image frame to obtain the aligned second image features. The aligned second image features are processed by the residual network and the fully connected layer in turn to obtain feature vectors of each category, where the number of feature vectors of each category is: the number of feature vectors of similar falsely detected targets output by the residual network and output by the fully connected layer, The feature vectors of each category are fused into a synthetic feature vector as a false detection target template, and stored in a false detection database.
