A bending machine visual safety protection method, system and readable storage medium

Abstract

This invention proposes a visual safety protection method, system, and readable storage medium for bending machines. The method includes: acquiring multi-view images of the bending machine's operating area, including top, left, and right views; extracting viewpoint features from the multi-view images based on a multi-stream neural network model to obtain top-view features and features from different lateral views; performing cross-view feature alignment and information fusion on the top-view features and features from different lateral views through a cross-attention fusion layer of the multi-stream neural network model to generate a unified fused feature; performing multi-task prediction in parallel based on the unified feature, the multi-task prediction including at least hand key point detection, mold key part detection, prediction of the minimum distance between the hand and the mold, and a risk probability value indicating a risk of crushing; and generating a safety intervention decision signal based on the risk probability value and / or the minimum distance. This invention can accurately identify dangerous actions of operators and improve operational safety.

Description

Technical Field

[0001] This invention relates to the field of industrial safety visual monitoring, and in particular to a visual safety protection method, system, and readable storage medium for bending machines. Background Technology

[0002] In bending machine processing systems, the safety protection devices typically use beam scanning or light curtain projection to form a protective barrier around the danger zone. When the beam is blocked, the system detects the presence of a foreign object and triggers an emergency stop. However, in actual processing, the workpiece often has a complex geometry, and its lateral protrusions or edges may accidentally extend into and block the protective beam. The safety protection function must be deactivated before normal operation, while the internal danger zone (such as the bending point) still poses a risk of personnel operation or accidental entry, thus leaving personnel without effective safety protection. This blind spot caused by workpiece lateral obstruction has become a prominent safety hazard in the field of bending processing. Summary of the Invention

[0003] This invention provides a visual safety protection method, system, and readable storage medium for bending machines to solve the problems existing in related technologies. The technical solution is as follows: In a first aspect, embodiments of the present invention provide a visual safety protection method for a bending machine, comprising: Collect multi-view images of the bending machine's operating area, including top view, left view, and right view; Based on a multi-stream neural network model containing multiple parallel processing branches, viewpoint features are extracted from multi-view images to obtain top-down viewpoint features and different side viewpoint features. By using the cross-attention fusion layer of the multi-stream neural network model, cross-view feature alignment and information fusion are performed on top-view features and features from different side views to generate unified features after fusion. Based on unified features, multi-task prediction is performed in parallel. Multi-task prediction includes at least hand key point detection, mold key part detection, minimum distance prediction between hand and mold, and risk classification. Among them, risk classification is used to output a risk probability value that represents the risk of extrusion in the current frame. Based on the risk probability value and / or minimum distance of the risk classification, a safety intervention decision signal is generated to trigger the safety control command of the bending machine.

[0004] In one embodiment, the multi-view images are captured by a top-view camera group installed directly above the bending machine and side-view camera groups installed on the left and right sides of the bending machine, respectively; the field of view of the top-view camera group and the field of view of the side-view camera group cover the operating area of ​​the bending machine, and are used to capture the operator's side posture and hand movement.

[0005] In one implementation, it further includes: Real-world working condition videos were acquired and uploaded to a cloud server. The cloud server was then used to annotate the real-world working condition videos frame by frame to obtain the original image training set. The annotated content included the hand contour, the forming end position of the upper die punch and its stamping movement path, the longitudinal center line of the lower die V-shaped groove and its groove width, the safety buffer area, and behavior labels. When constructing the training set, the cloud server is used to oversample the samples corresponding to rare events in the original image training set to balance the sample distribution and obtain a balanced dataset. Rare events include hand keypoints that are close to closed. Data augmentation is then performed on the images in the balanced dataset to obtain an augmented training set for model update and training. Data augmentation includes at least one of random occlusion, color perturbation, affine transformation, and mirror flipping.

[0006] In one implementation, the multi-stream neural network model includes: The keypoint detection head includes a first sub-network, which is used to extract and output the coordinates of specified hand keypoints from uniform features; The target detection head includes a second sub-network, which is used to locate the forming end position of the upper die punch and the slot position of the lower die V-shaped groove based on uniform features. The distance regression head includes a third sub-network, which is used to predict the minimum distance between the hand and the mold based on uniform features. The risk classification head includes a fourth sub-network. The input of the fourth sub-network is connected to the output of the key point detection head, the target detection head, and the distance regression head, respectively. It is used to generate risk probability values ​​based on the coordinates of the hand key points, the forming end position of the upper die punch, the slot position of the lower die V-shaped groove, and the minimum distance. The decision head includes a fifth sub-network. The input of the fifth sub-network is connected to the output of the key point detection head, target detection head, distance regression head, and risk classification head, respectively. It is used to generate a safety intervention decision signal based on the coordinates of the hand key points, the forming end position of the upper die punch, the slot position of the lower die V-shaped groove, the minimum distance, and the risk probability value. The safety intervention decision signal is used to indicate a safe or intervention-required state.

[0007] In one implementation, the training process of a multi-stream neural network model includes: Construct a multi-task weighted loss function, which includes a key point detection loss term, an object detection loss term, a distance regression loss term, and a risk classification loss term, and assign weights to each loss term to form the total loss. In each iteration, the gradient is calculated based on the total loss, and the parameters of the multi-stream neural network model are updated through backpropagation. During training, the weights of each loss term are dynamically adjusted according to the performance metrics of each task on the validation set to balance the convergence speed of each task.

[0008] In one implementation, it further includes: The trained multi-stream neural network model is sequentially pruned, quantized, and format-converted using model optimization tools to obtain a runtime format model adapted to the target CNC system platform. The quantization process includes training followed by quantization, and then quantization-aware training to convert the model to INT8 format. The format conversion process includes exporting the quantized model to the ONNX universal format and then converting it to a runtime format adapted to the RK3588 multi-core processor through a middleware engine.

[0009] In one implementation, it further includes: When the risk probability is within a specified range, or the output confidence of the key point detection head is lower than the preset value, the frame image is judged as an uncertain sample and uploaded to the cloud server. The cloud server periodically summarizes uncertain samples. When a new labeled uncertain sample data is detected, a training task is automatically triggered. The elastic weight solidification algorithm is used to incrementally train the current multi-stream neural network model to obtain a new model, and the new model is redeployed.

[0010] Secondly, embodiments of the present invention provide a visual safety protection system for a bending machine, comprising: The vision module is used to acquire multi-view images of the bending machine's operating area; The CNC module is connected to the vision module. The CNC module is equipped with a trained multi-stream neural network model, which is used to execute the vision safety protection method for bending machines as described above.

[0011] In one implementation, it further includes: The cloud server communicates with the CNC module to receive and aggregate uncertain samples. Based on the aggregated uncertain samples, it incrementally trains the multi-stream neural network model to generate an updated model. The updated model is then sent back to the CNC module.

[0012] Thirdly, embodiments of the present invention provide a computer-readable storage medium that stores a computer program, wherein when the computer program is run on a computer, the methods in any of the above-described embodiments are executed.

[0013] The advantages or beneficial effects of the above technical solutions include at least the following: This invention acquires multi-view images of the bending machine's operating area in real time, covering the entire danger zone with multi-view stereo vision. Even if the workpiece obscures a certain viewpoint, other views can still fully capture the state of the operating area, fundamentally avoiding detection failures caused by physical obstruction. This invention uses multiple parallel task heads in a multi-stream neural network model to accurately identify key points of the hand skeleton and combines this with a distance regression head to calculate the spatial distance between the hand and the bending point, thereby accurately distinguishing between the workpiece and the operator and avoiding safety logic vulnerabilities triggered by workpiece occlusion. Furthermore, it determines risk probabilities based on multi-task prediction and generates safety intervention decision signals, allowing for early response even before personnel enter the danger zone but with an approaching tendency, thus improving safety.

[0014] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the invention will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0015] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in the invention and should not be construed as limiting the scope of the invention.

[0016] Figure 1 This is a flowchart illustrating the visual safety protection method for bending machines according to the present invention. Figure 2 A three-dimensional structural diagram of a bending machine equipped with a top-view camera according to the present invention; Figure 3 This is a front view structural diagram of a bending machine equipped with a top-view camera according to the present invention. Figure 4 This is a side view of the bending machine with a top-view camera installed according to the present invention. Figure 5 This is a front view structural diagram of the bending machine equipped with three top-view cameras according to the present invention. Detailed Implementation

[0017] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0018] Example 1 This embodiment provides a visual safety protection method for bending machines, which can eliminate blind spots and improve detection reliability.

[0019] like Figure 1 As shown, the visual safety protection method for bending machines specifically includes the following steps: Step S1: Acquire multi-view images of the bending machine's operating area, including top view, left view, and right view.

[0020] The bending machine is equipped with a multi-view vision module, including at least three sets of fixed industrial cameras. The operating area of ​​the bending machine is captured by the multiple sets of fixed industrial cameras to obtain multi-view images.

[0021] It should be noted that the operating area of ​​the bending machine in this embodiment usually refers to the danger zone where the lower die of the bending machine is located and the space range that the operator's arm may touch during the operation.

[0022] In this embodiment, a top-view camera assembly is installed on the sheet metal outer casing directly above the bending machine to capture real-time images of the workbench area from a vertical downward perspective. The camera configuration varies depending on the bending machine model and the length of the workbench: like Figure 2 , Figure 3 , Figure 4 As shown, for small bending machines, i.e., when the workbench length is less than two meters, a single wide-angle overhead camera 1 is installed. The optical center of this camera is aligned with the longitudinal centerline of the workbench to ensure that the single camera's field of view can completely cover the entire workbench surface.

[0023] For medium-sized bending machines (i.e., worktable length greater than or equal to two meters but less than three meters), two top-view cameras should be installed. The first camera should be located at the front 1 / 3 of the worktable, and the second camera at the rear 2 / 3 of the worktable. During installation, the field of view of the two cameras needs to be adjusted to ensure that their horizontal overlap is ≥20%. Image stitching technology should be used to eliminate blind spots in the middle and achieve full coverage of the worktable.

[0024] like Figure 5 As shown, for large bending machines, i.e., when the workbench length is greater than or equal to three meters, three overhead cameras are installed. Each camera is equidistant from the workbench along its length, and the overlap rate of the field of view between adjacent cameras is controlled to be ≥15%. A complete overhead monitoring view is synthesized by stitching together multiple images.

[0025] To ensure image clarity and real-time performance, the camera resolution must be no less than 1920×1080, and the frame rate no less than 60fps, to capture high-speed moving mechanical parts and subtle hand movements of operators. Given the reflective nature of metal workpiece surfaces and the complex lighting conditions in the workshop, the camera must have built-in HDR (High Dynamic Range) functionality to balance overly bright and dark areas; it must also have low-light enhancement capabilities to ensure clear imaging in low-light or shadowy environments; and it should be equipped with an anti-reflective filter to suppress glare interference from metal sheets. The lens focal length needs to be adjusted according to the actual installation height of the camera, ensuring a projected pixel density of ≥2 pixels / mm on the workbench floor. This parameter guarantees sufficient pixel resolution for the smallest detection targets in the image (such as hands or small workpiece edges), providing a data foundation for subsequent algorithm recognition.

[0026] To compensate for the shortcomings of top-down cameras in capturing the operator's side posture and sensing the depth of hand movements, such as Figures 3-5 As shown, in this embodiment, side-view camera groups are installed on both sides of the workbench upright plate 300 to monitor the interaction status between the operator and the lower mold area from a side perspective.

[0027] like Figures 3-5 As shown, the side-view camera group includes a left-side camera 2 and a right-side camera 3, respectively installed on both sides of the bending machine's workbench upright plate 300, used to capture the operator's side posture, arm extension trajectory, and hand movements entering and exiting dangerous areas. To obtain the best field of view and avoid the operator's body obstructing the lower die area, such as... Figure 4 As shown, the optical axis of the top-view camera 1 lens is installed at an angle of 30° to 45° to the horizontal plane. This angle design can clearly capture the entire process of the hand entering the lower mold V-shaped groove from the outside of the machine body while ensuring that the operator's head and torso do not obstruct the view.

[0028] At the same time, such as Figure 3As shown, the combined field of view of all cameras must completely cover the entire length of the lower mold 200, as well as the opening and closing area between the upper mold 100 and the lower mold 200, while extending to the range of motion of the operator's arm during normal operation (typically including the area 0.5 meters in front of and to both sides of the workbench upright 300). Meanwhile, given that the ambient light on the side of the machine tool is often weaker than the front lighting, and the operator's body may cast shadows, the cameras need to have built-in or external infrared supplementary lighting modules that automatically switch or supplement light in low-light environments to ensure clear imaging in all weather conditions. For example, a ring of LED supplementary lights can be added around the cameras, with a color temperature set to 5000K to simulate natural light and avoid color cast. The supplementary lights are monitored in real time by a light sensor to monitor the ambient illuminance and automatically adjust the output power to ensure constant image brightness. Alternatively, the lens surface can be coated with an anti-fog and anti-oil optical coating to prevent cutting fluid vapor or dust from adhering and causing image blurring; at the same time, an air curtain or protective cover can be used to clean the lens surface regularly. Automatic white balance and dynamic contrast adjustment algorithms can also be executed by the built-in ISP of the camera or by the back-end processor to automatically adapt to color reproduction and brightness balance under different workshop lighting conditions, providing high-quality and highly consistent input images for deep learning models.

[0029] To ensure imaging stability and the durability of calibration parameters, all side-view cameras are rigidly fixed. Each camera is directly and rigidly locked to the bending machine tool structure (such as the vertical plate reinforcing rib or machine tool column) via a precision-machined metal bracket. They must not be installed on moving sliders, back gauges, or worktables to avoid field-of-view swaying or calibration failure caused by mechanical movement. The brackets must possess sufficient structural strength and shock-absorbing pads to suppress interference from high-frequency vibrations during stamping or bending processes on image acquisition.

[0030] To achieve spatial and temporal consistency of multi-view images and provide precisely aligned data for subsequent multi-stream neural network fusion processing, all cameras are connected to the same hardware trigger signal source. Global exposure is triggered by a TTL synchronization pulse emitted by the controller, ensuring that each camera captures images at the same time; the multi-channel signal trigger delay must be controlled within 10ms. For network cameras that do not support hardware triggering, the PTP network time synchronization protocol is used to ensure that the clock error of each camera is less than microseconds, achieving frame-level soft synchronization. The synchronously acquired video stream is transmitted in real time to the CNC system main control board via a gigabit Ethernet interface, where the image acquisition unit of the main control board performs timestamp alignment and buffering.

[0031] Through the above installation method, the side-view camera group and the top-view camera group form a multi-view stereo monitoring network, jointly acquiring multi-view images of the bending machine's operating area. In this embodiment, multi-view refers to both the top-view and side-view perspectives. The side-view perspective effectively solves the difficulty of hand recognition caused by workpiece obstruction or direct light in the top-view perspective. Infrared supplementary lighting and anti-fouling design ensure reliability in complex and harsh environments, while the high-precision synchronization mechanism provides a foundation for feature-level fusion of multiple images, significantly improving the accuracy of hand key point detection and risk distance calculation.

[0032] Step S2: Extract view features from multi-view images based on a multi-stream neural network model containing multiple parallel processing branches to obtain top-down view features and different side view features.

[0033] In this embodiment, a multi-stream neural network model is pre-built. During the training phase, this model requires the collection of video data from multiple scenarios. To cover various working conditions in actual production, a prototype system is deployed at a typical customer site, collecting no less than 1000 hours of real bending operation video. The data collection process covers the following combinations of variables: Sheet types: including cold-rolled steel, stainless steel, aluminum sheet, galvanized sheet and other materials; Plate thickness range: 0.5mm to 3mm; Bending angle: Covers any angle within the range of 0° to 180°; Light intensity: 100 lux to 10,000 lux, including environments with a mixture of natural and artificial light; Operators: include personnel of different heights, body types, clothing (long-sleeved / short-sleeved), and glove types (cotton, rubber, metal wire reinforced); Operational behaviors: covering the entire process of loading, positioning, intervention during bending, unloading, and abnormal intrusion.

[0034] The aforementioned real-world working condition videos were uploaded to a cloud server. The cloud server then annotated each frame of the videos, including the following annotations: Hand outline polygon: includes coordinates of key points for the five fingers; The forming end position (i.e., tip position) of the upper die punch and its stamping motion trajectory: with tip width information; The longitudinal center line and opening dimensions of the V-shaped groove in the lower mold: accurately mark the position and width of the V-shaped groove; Danger zone delineation: A safety buffer zone of ±10mm is defined with the upper and lower mold closure path as the center; Behavioral tags include categories such as safe operation, proximity to danger, unauthorized hand reach, and emergency avoidance.

[0035] To improve annotation efficiency, a semi-automatic annotation assistance mechanism can be introduced: first, the existing model performs "AI pre-annotation", and then manual correction is performed. This semi-automatic mode can improve annotation efficiency by more than 300%.

[0036] When constructing the training set, to address the problem of insufficient samples for rare events (such as the hand approaching the closed point of the upper and lower molds), the cloud server is used to oversample the samples corresponding to rare events in the original image training set to balance the sample distribution and obtain a balanced dataset; among them, rare events include the hand key points approaching the closed state.

[0037] Simultaneously, data augmentation processing is performed on the images in the balanced dataset to obtain an augmented training set for model update and training. This data augmentation includes at least one of random occlusion, color perturbation, affine transformation, and mirror flipping. The dataset is expanded through data augmentation and sample balancing to ensure that the final training dataset contains no fewer than 500,000 labeled images, with the ratio of positive to negative samples (safe / dangerous) controlled within 1:3.

[0038] The dataset constructed above is used to train a multi-stream neural network model, which includes a shared feature extraction backbone network and multiple task heads connected to this backbone network. To fully integrate top-down and side-view information, this embodiment uses a three-stream parallel convolutional neural network structure for the shared feature extraction backbone network, outputting input images from different perspectives respectively. The first stream is used to receive top-view images from a group of cameras above the bending machine. HRNet or Vision Transformer is used as the backbone network to extract high-resolution spatial features, preserving the fine structural information of the hands and molds to obtain top-view features.

[0039] The second stream is used to receive left-side view images from the camera on the left side of the workbench. ResNet-50 is used as the backbone network to extract left-side spatial features, focusing on the operator's left-side hand posture and entry / exit movements.

[0040] The third stream receives the right-side view image from the camera on the right side of the workbench. It also uses ResNet-50 as the backbone network to extract right-side spatial features, forming a symmetrical perception capability with the second stream.

[0041] Step S3: Through the cross-attention fusion layer of the multi-stream neural network model, cross-view feature alignment and information fusion are performed on top-view features and features from different side views to generate unified features after fusion.

[0042] A cross-attention fusion layer is introduced into the high-level feature space of the three-stream network to achieve cross-view feature alignment and information complementarity. The cross-attention fusion layer mainly projects the feature maps output by the three backbone networks into query matrices, key matrices, and value matrices, respectively. Through cross-view attention calculation, the features of each view can focus on the salient regions of other views, enhancing the feature representation of occluded or weakly textured regions. The unified features obtained after fusion are formed into a fused feature map, which simultaneously contains global layout information from the top view and depth contour information from the side view, providing a richer feature foundation for subsequent task heads.

[0043] On top of the fused features, multiple task heads are connected in parallel. Each task head shares the fused features but learns parameters independently. The task heads in this embodiment include: The key point detection head includes a first sub-network, which is used to extract and output the coordinates of specified hand key points from uniform features; in this embodiment, the two-dimensional coordinates of 15 hand key points are regressed.

[0044] The target detection head includes a second sub-network, which is used to locate the forming end position of the upper die punch and the slot position of the lower die V-shaped groove based on uniform features.

[0045] The distance regression head includes a third sub-network, which is used to predict the minimum distance D between the hand and the mold (upper mold tip or lower mold slot) based on uniform features.

[0046] The risk classification head includes a fourth sub-network. The input of the fourth sub-network is connected to the output of the key point detection head, the target detection head, and the distance regression head, respectively. It is used to generate a risk probability value P∈[0,1] based on the coordinates of the hand key points, the forming end position of the upper die punch, the slot position of the lower die V-shaped groove, and the minimum distance. This value is used to characterize the possibility of the squeezing risk occurring at the current moment.

[0047] The decision head includes a fifth sub-network. The input of the fifth sub-network is connected to the output of the key point detection head, target detection head, distance regression head, and risk classification head, respectively. It is used to generate a binary safety intervention decision signal S, S∈{0,1}, based on the coordinates of the hand key points, the forming end position of the upper die punch, the slot position of the lower die V-shaped groove, the minimum distance, and the risk probability value. Here, 0 represents safety and 1 represents intervention required. The safety intervention decision signal is used to indicate whether the state is safe or requires intervention.

[0048] This embodiment uses the training dataset constructed above to train the multi-stream neural network model of the above architecture. Simultaneously, a multi-task weighted loss function is constructed to perform end-to-end training of the multi-stream neural network model. The expression for the total loss in this embodiment is: ; in, For the keypoint detection loss term, mean squared error or L1 loss is used to measure the deviation between the predicted keypoints and the labeled keypoints. This is the target detection loss term, which includes the target classification loss and the bounding box regression loss; For distance regression loss, smoothed L1 loss is used to measure the deviation between the predicted distance and the true distance; For the risk classification loss term, binary cross-entropy loss is used to measure the accuracy of risk probability prediction. Weights are assigned to each of the above loss terms to form the total loss.

[0049] In each iteration, the gradient is calculated based on the total loss, and the parameters of the shared feature extraction backbone network and multiple task heads in the multi-stream neural network model are updated through backpropagation. During training, the weights α, β, γ, and δ are dynamically adjusted according to the performance indicators of each task on the validation set. When a task converges slowly or has low accuracy, its weight is appropriately increased to balance the progress of multi-task learning.

[0050] Furthermore, to enhance the generalization ability of multi-stream neural network models under complex conditions, a phased training strategy can be adopted. Specifically, in the first phase, simple scene samples are used for training, including samples with uniform lighting, no occlusion, and slow hand movements; in the second phase, after the model has initially converged, medium-complexity samples such as lighting changes and partial occlusion are introduced for further training; in the third phase, complex scene samples such as strong reflections, rapid movement, and rare dangerous events are finally introduced for fine-tuning, so that the model can gradually adapt to various challenges in real production environments.

[0051] To address the characteristics of video sequence input, spatiotemporal consistency constraints can be introduced to enhance motion continuity modeling. Specifically, an optical flow estimation network is used to calculate optical flow information between adjacent video frames, representing pixel-level motion patterns. This optical flow information is then used as an auxiliary input and spliced ​​or attention-based fusion features to enable the model to perceive the motion trends of the hand and the mold. Simultaneously, temporal smoothing constraints are applied to the keypoint prediction results of adjacent frames to suppress inter-frame jitter and improve the stability of pose estimation.

[0052] Through the aforementioned spatiotemporal consistency constraints, the model can effectively integrate the implicit motion patterns and temporal continuity in the video into feature learning, significantly improving the accuracy of recognizing fast-moving and complex interactive scenes.

[0053] In this embodiment, the training of the multi-stream neural network model is based on the DLCV deep learning platform and executed in a distributed manner on a GPU cluster. A single complete training cycle does not exceed 72 hours, ensuring rapid iterative optimization of the model. This embodiment optimizes the trained multi-stream neural network model via a cloud server and uses model optimization tools to perform lightweight processing on the trained model. The lightweighted model is then distributed to the corresponding CNC module of the bending machine, where it receives, loads, and runs the model for real-time inference.

[0054] The lightweight post-processing includes sequentially performing channel pruning, quantization, and format conversion on the trained multi-stream neural network model to obtain a runtime format model adapted to the target CNC system platform. Specifically, channel pruning of the trained multi-stream neural network model removes redundant convolution kernels, achieving a model compression rate of over 40%, and ensuring that the size of the pruned model does not exceed 80MB.

[0055] In this embodiment, the pruned model undergoes quantization processing, which includes training followed by quantization using PTQ and then quantization-aware training using QAT to convert the FP32 model to INT8 format while controlling the accuracy loss to be less than 2%. The format conversion includes exporting the quantized model to the ONNX universal format and then converting the ONNX format model to a runtime format adapted to the target CNC system platform through a middleware engine. The converted model is then deployed to the RK3588 multi-core processor, and the model inference speed reaches more than 30 frames per second.

[0056] It should be noted that the vision module in this embodiment also runs on the RK3588 multi-core processor, using the RT-Linux operating system. The vision module can be specified to run on a specific core of the RK3588 to avoid resource contention with the CNC system. The vision module communicates with the CNC system in real time, with a communication latency of less than 200µs.

[0057] Step S4: Based on unified features, perform multi-task prediction in parallel. Multi-task prediction includes at least hand key point detection, mold key part detection, minimum distance prediction between hand and mold, and risk classification. Among them, risk classification is used to output a risk probability value that represents the risk of extrusion in the current frame.

[0058] This embodiment imports multi-view images acquired in real time during the bending machine's operation into a lightweight multi-stream neural network model. Multiple parallel processing branches of the multi-stream neural network model extract viewpoint features from the multi-view images, obtaining top-view features and features from different side views. Then, through the cross-attention fusion layer of the multi-stream neural network model, cross-view feature alignment and information fusion are performed on the top-view features and features from different side views, generating a unified fused feature. By combining a three-stream parallel structure with cross-attention fusion, the occlusion problem under a single viewpoint is effectively solved, improving the robustness of detection for the hand and the mold.

[0059] By employing a keypoint detection head and a risk classification head within a multi-stream neural network model, key points of the hand skeleton can be accurately identified. Combined with a distance regression head, the spatial distance between the hand and the bending point is calculated, thus accurately distinguishing between the workpiece and the operator, preventing safety logic vulnerabilities triggered by workpiece occlusion. Traditional devices can only output binary "triggered / not triggered" signals, unable to predict potential risks. This embodiment outputs continuous risk probability values ​​through a risk classification head, combined with a decision head to generate safety intervention decision signals (early warning or emergency stop). This allows for proactive response even before personnel enter the danger zone but before they approach, transforming passive protection into active safety.

[0060] Step S5: Based on the risk probability value and / or minimum distance of the risk classification, generate a safety intervention decision signal to trigger the safety control command of the bending machine.

[0061] In this embodiment, the CNC module of the bending machine has a built-in safety intervention decision unit, which generates control commands based on the outputs of multiple task heads and preset safety rules. According to the safety protection requirements of the bending machine, the following four levels of risk thresholds and judgment conditions can be preset according to the table below: Table 1 Relationship between Risk Level and Threshold Setting Risk level Judgment conditions Response Action normal P<0.3 No action, continuous monitoring warn 0.3 ≤ P<0.7 and D<150mm The audible and visual alarm is triggered, and the HMI pops up a message saying "Pay attention to hand position". Warning 0.7 ≤ P<0.9 or D<80mm Automatically reduce the slider's downward speed to 30% of its original speed. Danger P ≥ 0.9 or D < 30mm or hand enters the closed area Immediately send an emergency stop command, disable the servo enable, and lock the machine tool's motion. Where P is the extrusion risk probability output by the risk classification head, with a value range of [0,1]; D is the minimum spatial distance between the hand and the mold (upper mold tip or lower mold slot) output by the regression head, in millimeters; the closed area refers to the critical area where the upper and lower molds are about to close, specifically defined as a spatial range extending 10mm forward and backward from the line connecting the current upper and lower mold positions.

[0062] The decision-making unit executes a graded response according to the above risk levels. Under normal conditions, the slider runs normally at the preset speed and stroke, and the system only performs continuous monitoring without triggering any intervention actions. Under warning conditions, the audible and visual alarm device is triggered (warning light flashes and buzzer sounds intermittently), and a pop-up window on the CNC system's human-machine interface prompts "Pay attention to hand position" to remind the operator to adjust their hand position, while the slider's running speed remains unchanged. Under early warning conditions, the servo drive automatically reduces the slider's downward speed to 30% of the original set speed to gain more reaction time, while maintaining the audible and visual alarm. Under dangerous conditions, an emergency stop is immediately triggered, the servo drive enable is cut off via a hard-wired signal, the mechanical braking device is activated, and the slider's current position is locked until it is manually reset before it can be restarted.

[0063] To improve decision reliability, a multi-parameter fusion judgment mechanism is adopted. When multiple judgment conditions are met simultaneously, the response is executed based on the highest risk level. When the risk probability P is inconsistent with the risk level judged by the distance D, the higher level is executed. The binary signal S output by the decision head is used as a redundancy check: if S=1, the hazard level response is directly triggered regardless of other parameters.

[0064] In this embodiment, the CNC module incorporates a lightweight 3D engine developed based on the OpenCASCADE geometry kernel, possessing real-time rendering, spatial transformation, collision detection, and trajectory drawing capabilities. The engine connects to the output of the aforementioned multi-stream neural network model, receiving and parsing the coordinate data and status information output by each task head in real time. It dynamically constructs visualization elements within the 3D scene, including the operator's hand movement trajectory line, the dynamic position model of the upper and lower molds, highlighting of hazardous areas, marking and logging of interference moments, and playback mode and historical event viewing.

[0065] In this embodiment, the CNC module incorporates an uncertainty detection unit during real-time inference to evaluate the output quality of each frame of the model. A frame is classified as an "uncertain sample" when any of the following conditions are met: The risk probability P output by the risk classification head is in the fuzzy range of [0.4, 0.6], indicating that the model is hesitant in judging the current risk status; The confidence level of the hand key points output by the key point detection head is less than 0.5, indicating that the model's recognition of hand pose is not reliable enough.

[0066] After being identified as an uncertain sample, video segments of 5 seconds each before and after the current frame are extracted (approximately 150 frames in total if the frame rate is 30fps), preserving the complete contextual information of the uncertain event. The extracted video segments and the corresponding model output data are encrypted to prevent data leakage or tampering during transmission. The encrypted data packets are then uploaded to the cloud server via industrial Ethernet or 5G / 4G network for further processing.

[0067] The cloud server is responsible for aggregating uncertain samples from multiple terminal devices and performing incremental training and update distribution of the model. Specifically, the cloud server aggregates uncertain samples uploaded by all clients within a preset period (such as every day at midnight or at a fixed time every week); the aggregated uncertain samples need to be manually corrected and labeled to form a high-quality incremental dataset with labels; the system monitors the labeled data storage directory in real time, and when newly added labeled data is detected, it automatically adds the batch of data to the training queue, triggering the incremental training task.

[0068] To prevent the model from forgetting previously acquired knowledge while learning new samples, the cloud training module employs the following strategies: Base model loading: The latest version of the currently deployed model is used as the pre-trained weights and loaded into the training environment; Elastic weight solidification: Incremental training is performed using an elastic weight solidification algorithm. This algorithm calculates the importance of each parameter in historical tasks and penalizes updates of important parameters when training new tasks, thereby suppressing drastic changes in model parameters while learning new samples and effectively preventing catastrophic forgetting; Training termination condition: Training stops when the accuracy on the validation set no longer improves or when the preset number of iterations is reached.

[0069] After incremental training, candidate models are generated. The performance of these new models is then evaluated on a test set containing both historical and new data. The evaluation requires an accuracy improvement of at least 1% compared to the current version, with no degradation in key metrics such as recall and F1-score. Once validated, the model files and related configuration files are packaged into a firmware update package and digitally signed to ensure its integrity and provenance. Finally, the cloud server pushes the update package to the CNC module of the bending machine via OTA service, resulting in continuous improvement in the model's performance in visual safety monitoring of the bending machine.

[0070] This embodiment is based on a multi-view vision and multi-task learning-based bending machine safety monitoring system. Compared with traditional beam scanning or light curtain protection devices, its advantages include: First, through visual intelligent analysis, it distinguishes fingers, hands, and gloves from workpieces. Even if the side of the workpiece obscures the safety detection area, safety detection and protection can still be performed normally, solving the problem that laser safety protection devices cannot provide effective protection when the side of the workpiece obscures the beam. Second, laser safety protection devices must be installed in the protection area directly below the upper die, requiring recalibration of the device when changing the die. The vision device has a wide-range planar detection capability, and the camera can be appropriately offset outward to avoid the disassembly area of ​​the upper die, so that the camera does not need to be moved when changing the upper die. Third, the vision safety protection device can intelligently analyze and process different workpiece shapes without switching modes, improving work efficiency and avoiding losses and injuries caused by switching errors.

[0071] In summary, this embodiment systematically solves the safety blind spot problem caused by workpiece occlusion in traditional laser safety protection devices from multiple levels, including perception dimension, identification capability, decision-making logic, and engineering adaptability, and significantly improves the level of personnel safety protection in laser processing scenarios.

[0072] Example 2 This embodiment provides a vision safety protection system for a bending machine, which includes at least a vision module, a CNC module, and a cloud server.

[0073] The vision module is used to acquire multi-view images of the bending machine's operating area.

[0074] The vision module includes a top-view camera group installed directly above the bending machine and side-view camera groups installed on the left and right sides of the bending machine, respectively. The field of view of the top-view camera group and the field of view of the side-view camera group cover the operating area of ​​the bending machine and are used to capture the operator's side posture and hand movements.

[0075] The CNC module is connected to the vision module. The CNC module is equipped with a trained multi-stream neural network model, which is used to execute the bending machine vision safety protection method described above, so as to provide warnings of dangerous situations while controlling the bending machine to perform workpiece bending operations.

[0076] The cloud server communicates with the vision module and CNC module to receive and aggregate uncertain samples. Based on the aggregated uncertain samples, it incrementally trains the multi-stream neural network model to generate an updated model. The updated model is then distributed to the CNC module to improve the model's adaptability to complex working conditions, personnel differences, and environmental changes, thereby achieving accuracy stability and performance evolution of the safety monitoring system throughout its entire lifecycle.

[0077] It should be noted that the functions of each module in the system of this embodiment can be found in the corresponding descriptions in the above methods, and will not be repeated here.

[0078] Example 3 This invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this invention.

[0079] This invention also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device on which the chip is installed to perform the method provided in this invention.

[0080] This invention also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in this invention.

[0081] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting the Advanced Reduced Instruction Set Computing (RISC) machine (ARM) architecture.

[0082] Further, optionally, the aforementioned memory may include read-only memory and random access memory, and may also include non-volatile random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0083] In the above embodiments, implementation can be achieved, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0084] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0085] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0086] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in the present invention, and these should all be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual safety protection method for a bending machine, characterized in that, include: Collect multi-view images of the bending machine's operating area, including a top view, a left view, and a right view; The multi-view images are subjected to view feature extraction based on a multi-stream neural network model containing multiple parallel processing branches, resulting in top-down view features and different side view features. The cross-attention fusion layer of the multi-stream neural network model performs cross-view feature alignment and information fusion on the top-view features and the features of different side views to generate a unified fused feature. Based on the unified features, multi-task prediction is performed in parallel. The multi-task prediction includes at least hand key point detection, mold key part detection, minimum distance prediction between hand and mold, and risk classification. The risk classification is used to output a risk probability value that represents the risk of squeezing in the current frame. Based on the risk probability value of the risk classification and / or the minimum distance, a safety intervention decision signal is generated to trigger the safety control command of the bending machine.

2. The visual safety protection method for bending machines according to claim 1, characterized in that, The multi-view images are captured by a top-view camera group installed directly above the bending machine and side-view camera groups installed on the left and right sides of the bending machine respectively; the field of view of the top-view camera group and the field of view of the side-view camera group cover the operating area of ​​the bending machine, and are used to capture the operator's side posture and hand movement.

3. The visual safety protection method for bending machines according to claim 1, characterized in that, Also includes: Real-world working condition videos are acquired and uploaded to a cloud server. The cloud server is then used to annotate the real-world working condition videos frame by frame to obtain an original image training set. The annotated content includes the hand contour, the forming end position of the upper die punch and its stamping movement path, the longitudinal center line of the lower die V-shaped groove and its groove width, the safety buffer area, and behavior labels. When constructing the training set, the cloud server is used to oversample the samples corresponding to rare events in the original image training set to balance the sample distribution and obtain a balanced dataset; wherein, the rare events include hand key points that are close to closed; and the images in the balanced dataset are subjected to data augmentation processing to obtain an augmented training set for model update training; wherein, the data augmentation processing includes at least one of random occlusion, color perturbation, affine transformation and mirror flipping.

4. The visual safety protection method for bending machines according to claim 1, characterized in that, The multi-stream neural network model includes: The key point detection head includes a first sub-network, which is used to extract and output specified hand key point coordinates from the uniform features; The target detection head includes a second sub-network, which is used to locate the forming end position of the upper die punch and the slot position of the lower die V-shaped groove according to the unified features. The distance regression head includes a third sub-network, which is used to predict the minimum distance between the hand and the mold based on the unified features; The risk classification head includes a fourth sub-network. The input of the fourth sub-network is connected to the output of the key point detection head, the target detection head, and the distance regression head, respectively. It is used to generate a risk probability value based on the coordinates of the hand key points, the forming end position of the upper die punch, the slot position of the lower die V-shaped groove, and the minimum distance. The decision head includes a fifth sub-network. The input of the fifth sub-network is connected to the output of the key point detection head, the target detection head, the distance regression head, and the risk classification head, respectively. It is used to generate a safety intervention decision signal based on the coordinates of the hand key points, the forming end position of the upper die punch, the slot position of the lower die V-shaped groove, the minimum distance, and the risk probability value. The safety intervention decision signal is used to indicate a safe or intervention-required state.

5. The visual safety protection method for bending machines according to claim 1, characterized in that, The training process of the multi-stream neural network model includes: A multi-task weighted loss function is constructed, which includes a key point detection loss term, an object detection loss term, a distance regression loss term, and a risk classification loss term. Weights are assigned to each loss term to form the total loss. In each iteration, the gradient is calculated based on the total loss, and the parameters of the multi-stream neural network model are updated through backpropagation; and during training, the weights of each loss term are dynamically adjusted according to the performance metrics of each task on the validation set to balance the convergence speed of each task.

6. The visual safety protection method for bending machines according to claim 1, characterized in that, Also includes: The trained multi-stream neural network model is sequentially subjected to channel pruning, quantization, and format conversion using model optimization tools to obtain a runtime format model adapted to the target CNC system platform. The quantization process includes training followed by quantization, and then quantization-aware training to convert the model into INT8 format. The format conversion includes exporting the quantized model to the ONNX universal format and then converting it into a runtime format adapted to the RK3588 multi-core processor through a middleware engine.

7. The visual safety protection method for bending machines according to claim 1, characterized in that, Also includes: When the risk probability is within a specified range, or when the output confidence of the key point detection head is lower than a preset value, the current frame image is determined to be an uncertain sample, and the uncertain sample is uploaded to the cloud server. The cloud server periodically summarizes uncertain samples. When a new labeled uncertain sample data is detected, a training task is automatically triggered. The elastic weight solidification algorithm is used to incrementally train the current multi-stream neural network model to obtain a new model, and the new model is redeployed.

8. A visual safety protection system for a bending machine, characterized in that, include: The vision module is used to acquire multi-view images of the bending machine's operating area; A numerical control module is connected to the vision module, and the numerical control module is equipped with a trained multi-stream neural network model for executing the bending machine vision safety protection method as described in any one of claims 1 to 7.

9. The visual safety protection system for bending machines according to claim 8, characterized in that, Also includes: A cloud server, which is connected to the CNC module, is used to receive and aggregate uncertain samples, perform incremental training on the multi-stream neural network model based on the aggregated uncertain samples, generate an updated model, and then send the updated model back to the CNC module.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the visual safety protection method for bending machines as described in any one of claims 1 to 7.

Images