Intruder monitoring method and system based on laser and video image fusion

Abstract

The invention relates to the field of image recognition, in particular to an invader monitoring method and system based on laser and video image fusion. The method comprises the following steps that real-time laser scanning monitoring signals are collected based on a multi-array laser, dynamic area laser shielding disturbance excavation is carried out, and a working area foreign matter light beam shielding disturbance map is constructed; the method comprises the following steps: acquiring a working area monitoring video stream according to a high-definition camera, performing adaptive cache optimization and foreign matter invasion key frame segmentation, and extracting a foreign matter invasion video frame sequence; carrying out light beam shielding density calculation according to the light beam shielding disturbance map, and carrying out three-dimensional morphological analysis on the foreign matter invasion video frame sequence to obtain foreign matter three-dimensional morphological characteristics; and performing dynamic optical flow tracking and vector change frequency analysis on the foreign matter invasion video frame sequence, and constructing a foreign matter disturbance trajectory vector field. According to the invention, efficient and accurate invader monitoring is realized, and the operation safety and quality are improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to an intrusion monitoring method and system based on the fusion of laser and video images. Background Art

[0002] With the continuous improvement of the level of intelligent manufacturing and industrial automation, the palletizer, as one of the core equipment of intelligent logistics and production lines, has been widely used in many fields such as food processing, warehousing and transportation, electronic manufacturing, building materials and chemical industries. The palletizer precisely stacks and transports products through an automated control system, effectively improving the operation efficiency, reducing the labor cost, and enhancing the flexibility and stability of the production line. Especially in scenarios such as digital workshops and unmanned warehouses, the palletizer has become a key equipment to ensure the continuity of production and the efficiency of logistics.

[0003] However, during the high-speed operation and long-term operation of the palletizer, the operation area may face safety risks such as foreign objects entering by mistake, personnel leaning by mistake, and interference of the robotic arm. Once these intrusion behaviors are not identified and processed in time, they will not only cause equipment failures and product damage, but may also lead to operation interruptions and even pose a threat to the life safety of operators. Therefore, building a set of efficient, intelligent, and real-time intrusion monitoring and early warning mechanism has become an important technical requirement to ensure the safe and stable operation of the palletizing system.

[0004] Currently, the mainstream safety monitoring means of palletizers mostly rely on single sensors such as infrared pairs, safety light curtains, and ultrasonic detectors for area protection. Although these methods have the function of intrusion detection to a certain extent, they have many limitations. In addition, with the increasing complexity of industrial scenarios, traditional safety protection methods are also difficult to adapt to the dynamic and changing working environment. To address the above challenges, more efficient and intelligent intrusion monitoring and analysis methods are needed. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention proposes an intrusion monitoring method and system based on the fusion of laser and video images to solve at least one of the above technical problems.

[0006] To achieve the above object, the present invention provides an intrusion monitoring method based on the fusion of laser and video images, including the following steps: Step S1: Collect real-time laser scanning monitoring signals based on a multi-array laser, and conduct dynamic area laser occlusion disturbance mining to construct a foreign object beam occlusion disturbance map of the operation area; Step S2: Obtain the monitoring video stream of the operation area according to a high-definition camera, and conduct adaptive cache optimization and key frame segmentation of foreign object intrusion to extract the foreign object intrusion video frame sequence; Step S3: Calculate the beam shielding density according to the beam shielding perturbation map, and perform three-dimensional morphological analysis on the foreign object intrusion video frame sequence to obtain the three-dimensional morphological characteristics of the foreign object; Step S4: Perform dynamic optical flow tracking and vector change frequency analysis on the foreign object intrusion video frame sequence to construct a foreign object perturbation trajectory vector field; Step S5: Evolve the trend of the foreign object trajectory in the foreign object perturbation trajectory vector field, and perform foreign object trajectory conflict prediction to extract potential conflict path points; Step S6: Deduce the conflict area based on the potential conflict path points and the three-dimensional morphological characteristics of the foreign object, and make an adaptive early warning decision to generate an adaptive decision-making strategy for foreign object intrusion.

[0007] The high-frequency scanning signal obtained by the multi-array laser in the present invention can achieve high-precision positioning and change perception of any occlusion behavior in the space of the palletizing area. The mutation response of the laser frequency perturbation change is fast, and it can identify intrusion behaviors within milliseconds, especially suitable for areas of high-speed operating equipment. By constructing a "beam shielding perturbation map", the original signal can be structured and visualized, providing a physical reference for subsequent image processing and effectively improving the overall recognition accuracy of the system. Through adaptive caching and frame segmentation, only the video frame sequence highly correlated with the laser perturbation is extracted, effectively removing redundant images and reducing the processing load. By using the perturbation time sequence of the laser map to assist in determining the key video frames, "heterogeneous collaboration" between the video and laser information is achieved, and a fusion link is constructed. The cached and optimized video stream is more stable in image quality and frame rate, effectively reducing misjudgments or missed judgments caused by frame jitter and image blurring. Taking the shielding density (i.e., the spatial density of the beam being blocked) as the estimation basis for the volume and surface area of an object, a rough shape inversion is realized. Based on the density map to assist image analysis for edge restoration and three-dimensional modeling, the accuracy and robustness of shape recognition can be greatly improved. Identifying three-dimensional shapes such as long strips, blocks, and rolls helps with subsequent path prediction and collision volume simulation. Through optical flow tracking technology, the position, displacement direction, and speed changes of foreign objects in the video image can be captured in real time. Vectorizing the movement information in the time dimension provides a complete input for trajectory modeling and trend prediction. For example, whether approaching the equipment, entering the operation path, staying, or detouring can be used to optimize the pre-warning judgment strategy in advance. Through the evolution of the trajectory vector trend, the possible future movement paths of foreign objects can be predicted at multiple time points, achieving a leap from "recognition" to "prediction". Matching the predicted path with the equipment operation path in space, locking in possible conflict areas in advance, and effectively avoiding sudden collision situations. It can be accurate to "which second" and "which position" may have an intersection, greatly improving the time sensitivity and position accuracy of the pre-warning system. Combining the path points with the three-dimensional shape for conflict area deduction, considering the volume of the foreign object, movement trend, and the passing width of the operating arm, a risk assessment in the real scenario is realized. According to dimensions such as overlap probability, speed matching degree, and shape complexity, a risk level matrix is established, and a quantitative risk result is output. Automatically adjusting the pre-warning strategy according to the risk level, treating low interference, potential intrusion, and high-risk collisions differently, and truly achieving an "adaptive, hierarchical, and intelligent" response. Only triggering a severe response mechanism for high-risk path points, effectively reducing interference actions such as shutdown and speed reduction caused by false alarms, and improving the economy and practicality of the system.

[0008] In this specification, a monitoring system for intruding objects based on the fusion of laser and video images is provided, which is used to execute the method for monitoring intruding objects based on the fusion of laser and video images as described above, and includes: A beam perturbation module, which is used to collect real-time laser scanning monitoring signals based on a multi-array laser, perform dynamic area laser shielding perturbation mining, and construct a foreign object beam shielding perturbation map of the working area; A video segmentation module, which is used to obtain a monitoring video stream of the working area according to a high-definition camera, perform adaptive cache optimization and key frame segmentation of foreign object intrusion, and extract a foreign object intrusion video frame sequence; A three-dimensional morphology module, which is used to calculate the beam shielding density according to the beam shielding perturbation map, and perform three-dimensional morphology analysis on the foreign object intrusion video frame sequence to obtain foreign object three-dimensional morphology characteristics; A dynamic optical flow tracking module, which is used to perform dynamic optical flow tracking and vector change frequency analysis on the foreign object intrusion video frame sequence, and construct a foreign object perturbation trajectory vector field; A conflict prediction module, which is used to perform the evolution of the foreign object trajectory trend on the foreign object perturbation trajectory vector field, perform foreign object trajectory conflict prediction, and extract potential conflict path points; An intrusion decision-making module, which is used to perform conflict area deduction based on potential conflict path points and foreign object three-dimensional morphology characteristics, and perform adaptive early warning decision-making to generate a foreign object intrusion adaptive decision-making strategy.

[0009] The present invention constructs a dense occlusion perturbation map through laser point cloud information, which helps to quickly lock the foreign object intrusion area in subsequent video images; the multi-array deployment eliminates the visual blind area and improves the recognition accuracy of different incident angles and irregular objects; it provides an independent signal source based on physical interference for the entire system, enhancing the multi-source data redundancy and anti-false alarm ability of the system. Through cache optimization, full-scale analysis is avoided, and only the key frames that may be involved in the intrusion are processed, reducing the computing power consumption; With the assistance of the laser map for positioning, the image analysis time is greatly shortened, and the real-time performance of video detection and positioning is improved; the cache adjustment and image enhancement technology can improve the recognition robustness under harsh conditions such as low illumination and strong backlight; after the video key frames are aligned with the laser perturbation map, the accurate definition of "when and where" the intruder is realized. By combining the laser shielding density and the image depth feature, the three-dimensional body shape, size and shape of the foreign object are accurately inverted; It can distinguish object types (such as people, cardboard boxes, robotic arms), adapt to differential safety responses in multiple scenarios; provide real geometric boundaries for subsequent trajectory prediction, facilitating the assessment of collision risks when objects move in space; the three-dimensional shape is used as the geometric parameter input for motion trend modeling, and the accuracy of the intrusion prediction model can be dynamically updated. Continuously restore the movement information of foreign objects at multiple time points to form a complete disturbance path; through vector frequency analysis, behavior patterns such as rapid crossing, repeated movement, and stationary lurking can be identified; the optical flow method has strong capture ability for fast-moving targets and can be widely applied to dynamic scenarios such as palletizing and logistics; the trajectory vector field is the core input of the subsequent conflict prediction algorithm to achieve path trend modeling. Based on the current trajectory trend, predict the positions of foreign objects at multiple future time points to achieve forward-looking safety control; it can compare and analyze the conflicts between the foreign object trajectory and the planned path of the palletizer, and output potential conflict points; not only judge "whether there is a conflict", but also quantify the conflict time window, overlap degree, and risk level; the prediction of conflict points enables the control system to reserve decision-making time such as braking, detouring, and pausing to avoid sudden interruptions. Combine the path overlap degree, motion trend, and three-dimensional volume to evaluate the potential hazard level of the intruder to the operation system; automatically select response methods such as "prompt warning, action slowdown, immediate stop" according to the risk level; it can be connected to systems such as the main control PLC and robot controller to achieve automatic obstacle avoidance or behavior interruption; the hierarchical warning mechanism can avoid unnecessary control actions triggered by minor interferences and ensure production continuity. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram of the step flow of a method for monitoring intruders based on the fusion of laser and video images according to the present invention; Figure 2 is a schematic diagram of the detailed implementation steps of step S1; Figure 3 is a schematic diagram of the detailed implementation steps of step S2; Figure 4 is a schematic diagram of the detailed implementation steps of step S3. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0011] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0012] This application example provides a method and system for monitoring intruders based on the fusion of laser and video images. The execution subjects of the method and system for monitoring intruders based on the fusion of laser and video images include but are not limited to: mechanical equipment, data processing platforms, cloud server nodes, network upload devices, etc. that carry this system, which can be regarded as general computing nodes of this application. The data processing platform includes but is not limited to: at least one of an audio and image management system, an information management system, and a cloud data management system.

[0013] Please refer to Figures 1 to 4 , the present invention provides an intrusion monitoring method based on the fusion of laser and video images. The intrusion monitoring method based on the fusion of laser and video images includes the following steps: Step S1: Collect real-time laser scanning monitoring signals based on a multi-array laser, and perform dynamic area laser shielding disturbance mining to construct a foreign object beam shielding disturbance map of the working area; Step S2: Obtain the monitoring video stream of the working area according to a high-definition camera, and perform adaptive cache optimization and key frame segmentation of foreign object intrusion to extract the foreign object intrusion video frame sequence; Step S3: Calculate the beam shielding density according to the beam shielding disturbance map, and perform three-dimensional morphology analysis on the foreign object intrusion video frame sequence to obtain the three-dimensional morphological characteristics of the foreign object; Step S4: Perform dynamic optical flow tracking and vector change frequency analysis on the foreign object intrusion video frame sequence to construct a foreign object disturbance trajectory vector field; Step S5: Evolve the foreign object trajectory trend of the foreign object disturbance trajectory vector field, and perform foreign object trajectory conflict prediction to extract potential conflict path points; Step S6: Deduce the conflict area based on the potential conflict path points and the three-dimensional morphological characteristics of the foreign object, and perform an adaptive early warning decision to generate an adaptive decision-making strategy for foreign object intrusion.

[0014] The high-frequency scanning signal obtained by the multi-array laser can realize high-precision positioning and change perception of any occlusion behavior in the stacking area space. The sudden change response of the laser frequency disturbance change is fast, and the intrusion behavior can be identified within the millisecond level, which is particularly suitable for high-speed equipment areas. By constructing a "beam shielding disturbance map", the original signal can be structured and imaged, so that the subsequent image processing has a physical reference, which effectively improves the overall recognition accuracy of the system. Through adaptive caching and frame segmentation, only the video frame sequence highly related to the laser disturbance is extracted, and the redundant pictures are effectively removed to reduce the processing load. The video key frame is determined by the disturbance timing of the laser map to achieve "heterogeneous collaboration" of video and laser information and build a fusion link. The video stream after cache optimization is more stable in image quality and frame rate, which effectively reduces misjudgment or missed judgment caused by frame jitter and image blur. The occlusion density (i.e., the spatial density of the beam being blocked) is used as the basis for estimating the volume and surface area of the object to achieve rough morphological inversion. Edge restoration and three-dimensional modeling based on dense map-assisted image analysis can greatly improve the accuracy and robustness of morphological recognition. Identifying three-dimensional shapes such as strips, blocks, and rolls is helpful for subsequent path prediction and collision volume simulation. Through optical flow tracking technology, the position, displacement direction, and speed change of foreign objects in video images can be captured in real time. Vectorizing the movement information in the time dimension provides complete input for trajectory modeling and trend prediction. Such as whether to approach the equipment, enter the operation path, stay or detour, etc., can be used for pre-optimization of early warning judgment strategies. Through the evolution of trajectory vector trends, the possible future movement paths of foreign objects can be predicted at multiple time points, realizing the transition from "identification" to "prediction". Spatially matching the predicted path with the equipment operation path, locking possible conflict areas in advance, and effectively avoiding sudden collisions. It can be accurate to "which second" and "which position" may cause intersections, greatly improving the time sensitivity and position accuracy of the early warning system. Combining path points with three-dimensional shapes to deduce conflict areas, taking into account the volume of foreign objects, movement trends, and the width of the operating arm, to achieve risk assessment in real scenarios. According to dimensions such as overlap probability, speed matching, and shape complexity, a risk level matrix is established to output quantitative risk results. The early warning strategy is automatically adjusted according to the risk level, and low interference, potential intrusion and high-risk collision are treated differently, truly achieving "adaptive, hierarchical, and intelligent" response. The heavy response mechanism is triggered only for high-risk path points, effectively reducing interruptions such as shutdowns and speed reductions caused by false alarms, and improving the economy and practicality of the system.

[0015] In the embodiment of the present invention, refer to Figure 1 , is a schematic flow chart of the steps of an intrusion monitoring method based on laser and video image fusion of the present invention. In this example, the steps of the intrusion monitoring method based on laser and video image fusion include: Step S1: Based on a multi-array laser, collect real-time laser scanning monitoring signals, conduct dynamic area laser shielding disturbance mining, and construct a foreign object beam shielding disturbance map of the operation area; In this embodiment, deploy multiple laser arrays within the operation area to achieve omnidirectional laser scanning monitoring. Each laser should have the same emission frequency and power to ensure data consistency and accuracy. Select a suitable laser (such as a near-infrared laser) to reduce the impact of ambient light and ensure effective operation under different lighting conditions. Ensure that the installation position of the laser can cover the entire operation area to avoid blind spots. When configuring the laser, consider its emission angle and distance to ensure that the intersection points of all laser beams can form dense point cloud data. Connect the laser to the data acquisition system to receive laser reflection signals in real time. Configure a highly sensitive photodetector to ensure accurate reception of weak laser reflection signals and connect to the computer system through a data acquisition card. Collect 1000 laser scans per second, and record the timestamp and intensity information of laser reflection for each scan. The generated point cloud data should include the spatial coordinates (X, Y, Z) and reflection intensity values of each point. Conduct dynamic analysis on the real-time obtained point cloud data to identify the areas where the laser beams are shielded. The number of shielded laser beams can be calculated by comparing the current point cloud data with the preset normal state point cloud data. Use a shielding detection algorithm (such as threshold segmentation or clustering analysis) to identify the shielding areas of the laser beams. Calculate the number of shielded laser beams in each area and generate a corresponding shielding map according to the coverage area. Integrate the mined shielding data to construct a foreign object beam shielding disturbance map within the operation area. The map should be able to clearly display the shielding degree of different areas and identify potential foreign object positions.

[0016] Step S2: Obtain the monitoring video stream of the operation area according to a high-definition camera, conduct adaptive cache optimization and key frame segmentation of foreign object intrusion, and extract the foreign object intrusion video frame sequence; In this embodiment, a high-definition camera is installed in the operation area to ensure that it can cover the entire area and has sufficient image resolution to clearly capture dynamic scenes. A camera with a wide viewing angle and high frame rate is selected to meet the requirements of real-time monitoring. The installation position of the camera should avoid direct sunlight and reflection interference to ensure normal operation under different lighting conditions. At the same time, appropriate protection measures are configured to ensure that the camera is not damaged in the operation environment. The high-definition camera is connected to the data processing system to ensure that the video stream can be transmitted to the computer for processing in real time. An efficient video transmission protocol (such as RTSP) is used to reduce latency and ensure stable data transmission. The camera is started to begin real-time acquisition of the monitoring video stream of the operation area. The system should be able to continuously record video data and mark it when a foreign object intrusion is detected for subsequent analysis. 30 frames of video are recorded per second to form a complete monitoring video stream. Ensure the integrity and coherence of the video stream for subsequent analysis and processing. According to the dynamic characteristics of the operation area, an adaptive caching mechanism is designed to optimize the delayed transmission of the video stream. This mechanism should be able to dynamically adjust the caching strategy according to the complexity of the video content and the network status to ensure smooth playback. Set the dynamic cache size and automatically adjust the cache length according to the complexity of the video stream and the network bandwidth to ensure smooth transmission in high-dynamic scenarios. The continuously acquired video stream is segmented into key frames, and the frame sequence related to foreign object intrusion is extracted. The key frames should be able to effectively represent the critical moments of foreign object intrusion to ensure that the extracted frames have important monitoring information. Use an algorithm based on motion detection to analyze the changes between consecutive frames in the video stream and identify the frames with significant changes as key frames. By analyzing the differences between consecutive frames, the key frames related to foreign object intrusion are extracted and recorded to form a foreign object intrusion video frame sequence.

[0017] Step S3: Calculate the beam shielding density according to the beam shielding perturbation map, and perform three-dimensional shape analysis on the foreign object intrusion video frame sequence to obtain the three-dimensional shape characteristics of the foreign object; In this embodiment, data of the occlusion region is extracted from the beam occlusion perturbation map. This map should contain the occlusion status of each laser beam and the corresponding intensity information. By analyzing this data, it is possible to identify which regions are affected by foreign object occlusion. Define the calculation formula for occlusion density. The occlusion density can be expressed as the ratio of the number of occluded beams to the total number of beams. Through calculation, the degree of beam occlusion in the working area can be evaluated. The number of occluded beams is statistically counted region by region, and the occlusion density of each region is calculated. By comparing the densities of different regions, potential risk regions can be identified. Collect the video frame sequence of foreign object intrusion. These frames should contain different perspectives of the foreign object in the working area. Ensure that the time interval of the video sequence is short to capture the morphological changes of the foreign object in real time. Select a suitable 3D reconstruction algorithm (such as structured light method or stereo vision method) to perform 3D morphological analysis on the foreign object. This algorithm should be able to generate a 3D model of the foreign object using multi-perspective images. Using the stereo vision method, by analyzing the images taken from different angles, the depth information of the foreign object is extracted. Process the video frames of the foreign object using the selected reconstruction algorithm to generate 3D point cloud data. Then, process the point cloud data to extract shape features (such as volume, surface area, contour, etc.). If the 3D reconstruction of the foreign object shows an irregular shape, calculate its surface area and volume to provide detailed morphological information.

[0018] Step S4: Perform dynamic optical flow tracking and vector change frequency analysis on the video frame sequence of foreign object intrusion to construct a foreign object perturbation trajectory vector field; In this embodiment, a suitable optical flow tracking algorithm (such as the Lucas-Kanade algorithm or the Horn-Schunck algorithm) is selected to analyze the foreign object intrusion video frame sequence. These algorithms can accurately track the movement of pixels in the video and are suitable for the analysis of fast-moving scenarios. Optical flow tracking is performed frame by frame on the foreign object intrusion video frame sequence. For each frame, the optical flow between the current frame and the previous frame needs to be calculated to obtain the movement information of the pixel points. First, stable corner points (using Shi-Tomasi corner detection) are selected as the tracking objects. Corner points are extracted from the first frame, the optical flow between the first frame and the second frame is calculated, the displacement information of each corner point is recorded, and updated to the current frame. The movement trajectory of each pixel point is recorded in the database and displayed through a visualization tool to help analyze the movement path and dynamic changes of the foreign object. Arrows are used to represent the movement direction and speed. After the optical flow tracking is completed, the optical flow vector information of each pixel point is extracted. These vectors contain the magnitude and direction of the movement, which are crucial for subsequent frequency analysis. The optical flow vector data of each pixel is recorded, including the horizontal component and the vertical component, to form a set of vector data sets. The calculation method for frequency analysis is determined. Usually, time series analysis techniques (such as Fourier transform or wavelet transform) are used to identify the change frequency of the optical flow vectors. This will help identify the movement change characteristics of the foreign object. Frequency analysis is performed on the recorded optical flow vector information to calculate its change frequency and identify the laws and characteristics of the movement. During the analysis process, the focus is on the frequency and amplitude of the speed change. If obvious periodic fluctuations are shown in the frequency analysis over a period of time, record the main frequency components and their corresponding amplitudes to help understand the dynamic behavior of the foreign object. Based on the optical flow vectors and their change frequencies, a foreign object perturbation trajectory vector field is constructed. The vector of each pixel point should represent the movement of the foreign object at that position, forming a complete vector field.

[0019] Step S5: Perform the evolution of the foreign object trajectory trend on the foreign object perturbation trajectory vector field, and perform foreign object trajectory conflict prediction, and extract potential conflict path points; In this embodiment, suitable trend analysis methods, such as time series analysis and polynomial fitting, are selected to identify and predict the changing trend of the foreign object trajectory. By analyzing the historical trajectory data, the motion patterns and changing characteristics can be extracted. The linear regression model or moving average method is used to calculate the changing trend of the foreign object position and identify the accelerating or decelerating motion states. The collected motion data is analyzed to calculate the characteristics such as the motion direction, speed change, and acceleration of the foreign object in different time periods. Through these characteristics, the motion trend can be identified. By calculating the position change of the foreign object in the past few seconds, if it is found that its speed gradually increases, it is marked as "accelerated movement", and the relevant data is recorded for subsequent analysis. According to the trend information of the foreign object trajectory, a suitable conflict prediction model is selected. A distance-based conflict detection method can be adopted to predict potential conflict points in combination with the real-time trajectory data. A safety distance threshold is set, for example, 0.5 meters. If the predicted trajectory of the foreign object and the operation path of the palletizer are within this distance, it is determined as a potential conflict. The predicted trajectory of the foreign object and the operation path are analyzed point by point to identify potential conflict path points. Spatial analysis techniques are used to judge the overlapping situation between the foreign object trajectory and the operation path. During the trajectory prediction process, if the distance between the foreign object trajectory and the palletizer path at a certain time point is less than the set safety distance, this point is recorded as a potential conflict path point.

[0020] Step S6: Based on the potential conflict path points and the three-dimensional morphological characteristics of the foreign object, conduct conflict area deduction and make an adaptive early warning decision to generate an adaptive decision strategy for foreign object intrusion.

[0021] In this embodiment, a conflict area model is constructed based on the three-dimensional morphological characteristics of the foreign object and potential conflict path points. This model should include the spatial occupancy information of the foreign object near the operation path. Combining the three-dimensional morphological characteristics, the range of the conflict area is deduced. Using a geometric model, the occupied space of the foreign object near the path is calculated according to its volume and shape. By combining the safety distance threshold, the boundary of the conflict area is set. Spatial analysis is performed on potential conflict path points to determine which path points will overlap with the operation path. By calculating the intersection of the three-dimensional occupancy model of the foreign object and the operation path, the possible conflict area is deduced. If the distance between a path point and the operation path is less than the set safety distance, the path point and its surrounding area are marked as the conflict area. Based on the deduction result of the conflict area, the criteria for early warning decision-making are set. The early warning criteria should combine the dynamic characteristics of the foreign object and the safety requirements of the operation path to ensure timely and effective response. Conflict area area: When the set area is greater than a specific threshold (such as 1 square meter), an early warning is triggered. Foreign object speed: If the speed of the foreign object in the conflict area exceeds the set threshold (such as 0.5 m / s), the early warning level is increased. According to the real-time monitoring data and the analysis result of the conflict area, an adaptive decision-making strategy is generated. When the conflict risk reaches the preset standard, the system should automatically generate corresponding countermeasures. If the conflict risk is high, the system can recommend reducing the speed of the palletizer, pausing the operation, or sending an alarm to notify the operator. After the adaptive decision-making strategy is generated, the system should immediately implement the corresponding measures and monitor the implementation effect. If the conflict risk decreases, the early warning level is adjusted according to the situation; if the conflict risk still exists, a high alert state is maintained continuously.

[0022] In this embodiment, refer to Figure 2 , which is a schematic diagram of the detailed implementation steps of step S1. In this embodiment, the detailed implementation steps of the said step S1 include: Continuously perform point cloud scanning on the operation area of the palletizer by a multi-array laser, and collect real-time laser scanning monitoring signals; Perform equal-length time series division according to the real-time laser scanning monitoring signals to obtain multiple laser scanning time windows; Perform laser frequency continuity identification on the multiple laser scanning time windows to generate a laser frequency continuous steady-state curve for each time window; Perform transient mutation detection according to the laser frequency continuous steady-state curve, and mark transient frequency mutation points; Calculate the frequency perturbation intensity and velocity gradient of the transient frequency mutation points, and perform dynamic area laser occlusion perturbation mining to construct a foreign object beam occlusion perturbation map of the operation area.

[0023] In this embodiment, multiple array lasers are configured within the working area of the palletizer to achieve omnidirectional laser scanning. The operating parameters of each laser (such as wavelength, emission frequency, emission power, etc.) should be adjusted according to the operating environment and monitoring requirements. The emission frequency of the laser is set to 1000 Hz to ensure rapid acquisition of high-resolution point cloud data. A real-time laser scanning monitoring system is constructed, including a laser emitter and a receiver, as well as a data acquisition and processing module. Ensure that the system can process laser reflection signals in real time and generate point cloud data. A high-sensitivity photodetector is selected to ensure accurate reception of reflection signals even under strong light interference, and a connection to the computer is achieved through a data acquisition card. The laser scanning system is started to continuously collect laser scanning signals within the working area and generate point cloud data. Appropriate algorithms are used to convert the laser reflection signals into three-dimensional coordinate point clouds to form a real-time monitoring model of the working area. The laser scanning results are recorded once every 1 second to form time series data for subsequent analysis. According to the timestamps of the real-time laser scanning data, the collected data is divided into equally long time series. The length of the division should be reasonably configured according to the dynamics of the operating scenario and the laser scanning frequency. The length of each time window is set to 10 seconds to ensure that dynamic changes within the working area can be captured. The laser scanning data is sliced, and the entire data set is divided into multiple equally long time windows. Each window should contain the same number of laser scanning data points for subsequent analysis. If the total number of collected data points is 10,000, the corresponding number of data is extracted in each 10-second window to keep the data volume within each window consistent. The data of multiple divided laser scanning time windows is recorded in the database and preliminarily analyzed to ensure the integrity and usability of the data in each time window. The number of laser scanning data points and the fluctuations within each window are counted to confirm the data quality and ensure the reliability of subsequent analysis. Define the criteria for identifying laser frequency continuity to evaluate the laser frequency changes within each time window. Frequency continuity generally requires that within a certain time range, the amplitude of frequency change should remain within the set threshold. The set amplitude of frequency change less than ±0.5 Hz is defined as the continuous state. The laser frequency continuity is identified for the laser data within each time window, the corresponding laser frequency is calculated and compared with the set threshold to generate the laser frequency continuous steady-state curve for each time window. The frequency distribution within each time window is analyzed through Fourier transform to extract the main frequency components and form a frequency curve. The generated laser frequency continuous steady-state curve is recorded in the database and displayed through a graphical tool to analyze the trend of frequency changes. A line chart is generated to show the frequency changes in each time window to help identify the stability and fluctuations of the frequency. Define the detection criteria for transient mutations, which generally refer to points where the laser frequency changes significantly within a short period of time. The identification of mutation points should be based on the rate and amplitude of frequency change. The set mutation criterion is that the amplitude of frequency change exceeds ±1 Hz and the duration is less than 1 second.Perform transient mutation detection on the continuous steady-state curve of the laser frequency to identify frequency mutation points that meet the mutation criteria. Automatic identification can be achieved by setting up a detection algorithm. Traverse the frequency curve within each time window. If a point meets the mutation conditions, record the time, frequency value, and change amplitude of that point. Record the identified transient frequency mutation points in the database and display them through a visualization tool to help analyze the characteristics of the mutation and its impact on the working area. Generate an annotation map to mark the mutation points in the frequency curve graph for intuitive understanding of the key positions of frequency fluctuations. Define a calculation method for the frequency perturbation intensity, which is usually expressed as the frequency difference before and after the frequency mutation point changes. The velocity gradient represents the speed of frequency change. The formula for calculating the perturbation intensity is: perturbation intensity = |frequency after mutation - frequency before mutation|, and the velocity gradient is calculated as: gradient = perturbation intensity / time interval. Calculate the perturbation intensity and velocity gradient for the identified transient frequency mutation points. Ensure that each mutation point has corresponding frequency values and timestamps for accurate calculation. If the frequency before mutation is 20 Hz, the frequency after mutation is 22 Hz, and the time interval is 0.5 seconds, then the perturbation intensity is 2 Hz and the velocity gradient is 4 Hz / s. Define the laser shielding perturbation in the dynamic area based on the frequency mutation points and their perturbation intensities. Shielding perturbation usually refers to the change in laser signal reflection caused by external factors (such as foreign objects, obstacles, etc.). Set a mark for shielding perturbation when the frequency perturbation intensity exceeds a certain threshold (such as 1 Hz). Based on the identified transient frequency mutation points and their related data, construct a foreign object beam shielding perturbation map of the working area. This map should be able to reflect the laser signal changes at different time and space positions. Use a three-dimensional visualization tool to plot the shielding perturbation data into a map to display the laser signal intensity changes in different areas. Record the constructed foreign object beam shielding perturbation map in the database and display it through a visualization tool for analyzing the monitoring effect of the working area. Generate a heat map to show the laser shielding conditions at different positions in the working area to help identify potential interference sources.

[0024] In this embodiment, refer to Figure 3 , which is a schematic diagram of the detailed implementation steps of step S2. In this embodiment, the detailed implementation steps of step S2 include: Perform panoramic operation monitoring on the palletizing machine working area using a high-definition camera to obtain the monitoring video stream of the working area; Perform adaptive cache optimization on the monitoring video stream of the working area to construct a delay-optimized video stream; Calculate the beam shielding time length of the foreign object beam shielding perturbation map in the working area to obtain the intrusion behavior time curve; Locate the foreign object intrusion position based on the foreign object beam shielding perturbation map in the working area; Perform key frame segmentation of the foreign object intrusion on the latency-optimized video stream according to the foreign object intrusion location and the intrusion behavior time curve, and extract the foreign object intrusion video frame sequence.

[0025] In this embodiment, install a high-definition camera in the palletizer operation area to ensure that it has sufficient field of view coverage and resolution to clearly capture the dynamic situation in the operation area. The camera should be able to support video streams with a resolution of 1080p or higher to ensure image quality. Select a camera with a wide-angle lens to cover the entire operation area, and set the frame rate to 30fps to ensure the smoothness of the video stream. Build a video acquisition system, connect the camera to the data management system, and transmit the video stream data in real time. Ensure that the system has a stable network connection to avoid data loss. Connect the camera using Ethernet to ensure that the video stream can be transmitted to the server in real time for storage and processing. Start the camera to start real-time acquisition of the monitoring video stream in the operation area. The system should be able to continuously record video data and mark it when specific events occur for subsequent analysis. Save a segment of the video stream every 1 second to ensure data integrity and provide rich information for subsequent analysis. Design an adaptive caching mechanism according to the dynamic characteristics of the operation area to optimize the latency transmission of the video stream. This mechanism should be able to dynamically adjust the caching strategy according to changes in the network state and video content. Set the dynamic cache size and automatically adjust the cache length according to the complexity of the video stream and the network bandwidth to ensure smooth transmission in high-dynamic scenarios. Perform latency optimization processing on the acquired video stream to ensure that there is no obvious latency during real-time transmission. The streaming media transmission protocol (such as RTSP) can be used to reduce latency. Test different cache sizes and transmission parameters to find the best configuration to ensure effective reduction of latency while playing smoothly. Record the performance of the optimized latency video stream, including indicators such as latency time, packet loss rate, and video quality. Evaluate the optimization effect through experiments to ensure the stability and reliability of the video stream. Conduct network bandwidth tests and record the latency under different bandwidth conditions to evaluate the effect of the adaptive caching mechanism.

[0026] Based on the previously constructed light beam occlusion perturbation map, analyze the light beam occlusion situation in each time period. It is necessary to identify the duration of the light beam being occluded to evaluate the duration of the intrusion behavior. Set a threshold. When the occlusion signal lasts for more than 1 second, it is recorded as an occlusion event, and the duration of this event is calculated. Calculate the duration of each occlusion event to ensure that each event has accurate start and end times. Record these durations to generate the intrusion behavior time curve. By analyzing the timestamp data of the occlusion events, calculate the duration of each event and record it in the database. According to the light beam occlusion perturbation map and video stream data, set the positioning algorithm for the foreign object intrusion location. This algorithm should be able to combine laser and video information to accurately determine the location of the intrusion object. Use an object detection algorithm based on image processing, combined with the timestamp of the occlusion event, to locate the specific location of the foreign object. Analyze the video stream and light beam occlusion data to identify the specific location of the foreign object intrusion. It can be summarized by comparing the changes in the video stream and the data in the laser occlusion map. If a light beam occlusion is found at a certain time point and matches the object movement in the video stream, record this location as the foreign object intrusion location. Record the identified foreign object intrusion location in the database and display it through a visualization tool to help analyze the distribution of the intrusion objects. Mark the location of the foreign object in the video stream and generate a heat map to show the intrusion frequency and intensity at different locations. According to the intrusion behavior time curve and the foreign object intrusion location, set the criteria for key frame extraction.

[0027] The key frames should be able to represent the critical moments of the intrusion behavior to ensure that the extracted frames have important monitoring information. Set the frames within 1 second before and after the occurrence of the foreign object intrusion as key frames to capture the whole process of the intrusion behavior. Perform key frame segmentation on the latency-optimized video stream to extract the frame sequence related to the foreign object intrusion. Automatically extract key frames using timestamp data and positioning information. If the foreign object intrusion occurs at 10:00:05, extract all the frames between 10:00:04 and 10:00:06 as key frames. Record the extracted foreign object intrusion video frame sequence in the database and analyze it to ensure that each key frame can provide effective information. Generate a video clip to display the key frame sequence for subsequent monitoring and analysis to facilitate corresponding countermeasures.

[0028] In this embodiment, the specific steps for adaptively caching and optimizing the monitoring video stream of the operation area to construct a latency-optimized video stream are as follows: Perform single-frame decoding on the monitoring video stream of the operation area to extract the image frame sequence; Perform frame-by-frame image region segmentation on the image frame sequence to obtain multiple local regions of each image frame; Perform grayscale pixel calculation on the local regions to generate the grayscale histogram information of each region; Calculate the adaptive grayscale change coefficient based on the grayscale histogram information to generate a regional grayscale transformation function; Stretch the pixel grayscale values of the local region according to the regional grayscale transformation function to generate a brightness-optimized image; Traverse all the image frame sequences to construct a brightness-optimized image frame sequence; Calculate the inter-frame delay and encoding / decoding delay of the brightness-optimized image frame sequence; Calculate the global average delay based on the inter-frame delay and encoding / decoding delay; Perform adaptive cache optimization on the job area monitoring video stream based on the global average delay to construct a delay-optimized video stream.

[0029] In this embodiment, configure a video stream single-frame decoding system to extract an image frame sequence from the real-time monitoring video stream. Select a suitable decoder that supports efficient compression formats (such as H.264 or H.265) to ensure fast decoding and processing. Set the frame rate of the decoder to 30fps to ensure that 30 frames of images can be extracted per second to obtain sufficient image data for subsequent analysis. Start the video decoding system and read the video stream frame by frame. Each time an image frame is read, save it as an independent image file for subsequent processing. Extract the first frame to the 300th frame from the video stream to form an image frame sequence, ensuring that the quality of each frame of the image meets the analysis standard. Record the extracted image frame sequence in the database and perform a preliminary evaluation to ensure that each frame of the image is clear and undamaged. Analyze the extracted image frames through image quality evaluation metrics (such as PSNR, SSIM) to ensure that they meet the requirements for subsequent processing.

[0030] According to the characteristics of the image frames, select an appropriate image segmentation algorithm (such as threshold-based, edge detection, or region growing algorithm) to perform frame-by-frame image region segmentation. The selected algorithm should be able to effectively identify important regions in the image. Use the Canny edge detection algorithm to extract important edge information in the image for subsequent region segmentation. Apply the selected region segmentation algorithm to each frame of the image to identify multiple local regions. Each local region should be able to reflect different features or objects in the image. Segment a frame of the image to identify the foreground and background regions, obtaining multiple local regions for subsequent analysis. Determine the method for calculating gray pixels, usually by converting the color image to a grayscale image and then calculating the gray pixel values of each local region. Use the weighted average method to convert RGB values to gray values, with the calculation formula: Gray = 0.2989R + 0.5870G + 0.1140*B. Calculate the gray histogram for the grayscale image of each local region, recording the number of pixels at each gray level for analyzing the brightness distribution of the image. If the gray value range of a local region is 0 - 255, generate a histogram with 256 bins, recording the frequency of each gray value occurrence. Record the generated gray histogram information in the database and display it through a visualization tool to analyze the gray distribution of the image regions. Generate a histogram image to show the gray distribution of each local region for analyzing the brightness characteristics of the image. Define the calculation criteria for the adaptive gray change coefficient, usually calculated based on the statistical characteristics (such as mean, variance) of the gray histogram to determine the gray range that needs to be adjusted. Set the calculation formula for the gray change coefficient as: change coefficient = (maximum gray - minimum gray) / (maximum gray + minimum gray). According to the gray histogram of each local region, calculate the corresponding adaptive gray change coefficient to generate a regional gray transformation function. If the maximum gray of a local region is 200 and the minimum gray is 50, the change coefficient is (200 - 50) / (200 + 50) = 0.6. Determine the method for gray value stretching, usually by expanding the gray values through a linear transformation formula to optimize the brightness and contrast of the image. Set the gray stretching formula as: new gray = change coefficient * (original gray - minimum gray) + minimum gray. Apply the gray value stretching process to each local region to generate a brightness-optimized image. Ensure that after the gray values of each region are stretched, the visual effect of the image can be significantly improved. If the original gray value of a local region is 100 and the new gray value after stretching is 120, update the pixel values of that region. Traverse all the extracted image frame sequences, perform brightness optimization processing on each frame, and record the results as a new image frame sequence.

[0031] For the extracted 300 frames of images, apply gray-scale stretching optimization frame by frame to form a new sequence of brightness-optimized images. Store the generated sequence of brightness-optimized image frames in a database for subsequent analysis and use. Ensure that each frame of the image has good quality and usability. Establish a new folder structure to store the brightness-optimized image frames by serial number to ensure the neatness and easy access of the data. Conduct a quality assessment of the generated sequence of brightness-optimized image frames to ensure that the clarity, contrast, and brightness of the images meet the expected standards. Use image quality assessment methods (such as SSIM) to analyze the quality of the optimized images and confirm that they meet the monitoring requirements. Define the calculation methods for inter-frame delay and codec delay to evaluate the performance of the sequence of brightness-optimized image frames. The inter-frame delay is usually the time difference between adjacent frames, while the codec delay is the total time for video stream processing. Set the inter-frame delay as the time difference between the current frame and the previous frame, and the codec delay as the total time processed by the decoder. Calculate the delay for the sequence of brightness-optimized image frames, record the data of the codec delay and inter-frame delay for each frame for subsequent analysis. If the average inter-frame delay is found to be 33 ms and the codec delay is 50 ms in 30 frames, record these data for summary analysis. According to the calculated inter-frame delay and codec delay, calculate the global average delay to evaluate the overall performance of the video stream. Calculate the global average delay as (sum of inter-frame delays + codec delay) / total number of frames to obtain a comprehensive delay metric. According to the calculation result of the global average delay, set the criteria for adaptive cache optimization to ensure that the transmission delay of the video stream is within an acceptable range. Set the goal of cache optimization to achieve smooth video playback when the global average delay does not exceed 100 ms. According to the delay calculation results, adjust the cache size and transmission parameters to achieve adaptive optimization. It can be tested by simulating different network conditions and video stream characteristics. Experiment with different caching strategies to find the optimal cache size and transmission scheme to achieve the lowest delay and the best transmission effect.

[0032] In this embodiment, refer to Figure 4 , which is a schematic diagram of the detailed implementation steps of step S3. In this embodiment, the detailed implementation steps of the said step S3 include: Perform deep target visual detection on the video frame sequence of foreign object intrusion and mark the foreign object target nodes; Separate the bounding boxes according to the foreign object target nodes and extract the foreign object image bounding boxes; Perform edge contour recognition on the foreign object image bounding boxes and extract the foreign object edge contour lines; Calculate the beam occlusion density of the beam occlusion perturbation map of foreign objects in the working area to obtain the beam occlusion density of foreign object intrusion; Perform multi-scale spatial registration and three-dimensional morphology analysis according to the beam occlusion density and the foreign object edge contour lines to obtain the three-dimensional morphological characteristics of the foreign objects.

[0033] In this embodiment, a suitable deep learning object detection model (such as YOLO, Faster R-CNN, or SSD) is selected to detect the foreign object intrusion video frame sequence. These models should be fully trained to effectively identify foreign objects in the operation area. The pre-trained YOLOv5 model is used, which performs well on multiple object detection datasets and can quickly and accurately identify target objects. Object detection is performed frame by frame on the extracted foreign object intrusion video frame sequence. Each frame is input into the deep object detection model to obtain the detection results of foreign object targets, including the position, category, and confidence score of the targets. Suppose multiple foreign objects are detected in a certain frame, and the results output by the model include the position information (such as the upper left and lower right coordinates) of each target and the corresponding category confidence. According to the detection results, set the processing criteria for the bounding boxes to ensure that the extracted bounding boxes can accurately enclose the foreign objects. The bounding boxes should have sufficient edge extensibility for subsequent image analysis. Set the minimum and maximum area criteria for the bounding boxes to filter out too small or too large boxes. Separate and extract the bounding boxes for the detected foreign object target nodes. Crop the bounding box of each target to generate the corresponding foreign object image. If the bounding box of a certain target is from the upper left corner (50, 50) to the lower right corner (150, 150), then extract this area from the original frame as a new image. Select a suitable edge detection algorithm (such as Canny edge detection or Sobel operator) to identify the edge contours of the extracted foreign object image bounding box. This algorithm should be able to effectively extract the edge information in the image. Use the Canny edge detection algorithm, which performs excellently in dealing with noise and edge detection accuracy. Apply the edge detection algorithm to each extracted foreign object image to identify the edge contour lines in the image. This process should minimize noise interference to improve the accuracy of edge detection. Set the high and low thresholds of Canny edge detection to 100 and 200, and generate an edge image through algorithm processing. Determine the calculation method for the beam occlusion density. The occlusion density usually represents the ratio of the number of beams occluded by foreign objects to the total number of beams within a certain period of time. Set the calculation formula for the occlusion density as: density = number of occluded beams / total number of beams. Calculate the density of the beam occlusion data in the operation area. Combine the beam occlusion perturbation map, count the number of occluded beams in each area, and compare it with the total number of beams. If the total number of beams in a certain area is 1000 and the number of occluded beams is 300, then the beam occlusion density of this area is 0.3. Select a suitable multi-scale spatial registration algorithm (such as ICP algorithm or feature matching) to register the foreign object edge contour lines with the beam occlusion density data. This process should be able to effectively align data at different scales. Use a feature-based registration method to match by extracting edge features and occlusion density features. Register the foreign object edge contour lines with the beam occlusion density to ensure that they are in the same spatial coordinate system. This step should ensure the accuracy of the data for subsequent three-dimensional morphology analysis.By matching transformation parameters, align the edge contour line with the occlusion density information to form a unified data set. Based on the registered data, perform three-dimensional morphological analysis. Extract the three-dimensional morphological characteristics of the foreign object (such as volume, surface area, shape, etc.) for further analysis. Use a three-dimensional reconstruction algorithm to generate a three-dimensional model of the foreign object, and evaluate the volume and surface characteristics of the foreign object through volume calculation methods. Record the extracted three-dimensional morphological characteristics of the foreign object in the database, and display the three-dimensional model through a visualization tool to analyze the foreign object characteristics. Generate a three-dimensional visualization diagram to display the morphological characteristics of the foreign object and help understand the characteristics and possible impacts of the intrusion object.

[0034] In this embodiment, step S4 includes the following steps: Perform dynamic optical flow tracking on the foreign object intrusion video frame sequence to identify the pixel point movement trajectories of the frame sequence; Perform optical flow vector calculation on the pixel point movement trajectories to obtain the foreign object optical flow vector information; Calculate the magnitude and direction of the foreign object optical flow vector information to obtain the speed and movement direction of the foreign object; Perform vector change frequency analysis in the time dimension on the foreign object optical flow vector information to obtain the foreign object movement rate change characteristics; Perform comprehensive motion pattern evolution on the foreign object movement rate change characteristics, speed, and movement direction to construct a foreign object perturbation trajectory vector field.

[0035] In this embodiment, a suitable optical flow tracking algorithm (such as the Lucas-Kanade algorithm or the Horn-Schunck algorithm) is selected to extract the motion trajectories of pixel points from the foreign object intrusion video frame sequence. These algorithms can effectively track the motion of pixels in the video and are suitable for dynamic scenes. Using the Lucas-Kanade algorithm, which performs well in processing small-range motions and is suitable for the analysis of high-frame-rate video streams. Optical flow tracking is performed frame by frame on the extracted foreign object intrusion video frame sequence. For each frame, the optical flow between the current frame and the previous frame needs to be calculated to obtain the motion information of pixel points. Among 100 consecutive frames, the displacement of each corner point (such as using the Shi-Tomasi corner detection) between two frames is calculated frame by frame, and these displacements are recorded as motion trajectories. Define the calculation method of the optical flow vector. The optical flow vector is usually determined by the displacement amount and time interval of each pixel and can describe the motion of an object in the image. Set the calculation formula of the optical flow vector as: 𝑉 = Δ𝑥 / Δ𝑡, U = Δy / Δt, where Δx and Δy represent the displacements in the horizontal and vertical directions respectively. Calculate the optical flow vector for the motion trajectory of each pixel point to obtain the foreign object optical flow vector information. This information includes the velocity components of each pixel point and their corresponding timestamps. If a pixel point has a horizontal displacement of 10 pixels and a vertical displacement of 5 pixels within 0.03 seconds, then the optical flow vector of this pixel is (333.33, 166.67) pixels / second. Determine the calculation methods for velocity and motion direction. The magnitude of the velocity can be calculated by the magnitude of the optical flow vector, while the motion direction is determined by the angle of the vector. The velocity calculation formula is , and the direction can be calculated by 𝜃 = arctan(U / V). Calculate the velocity and direction for each optical flow vector to obtain the motion velocity and direction information of the foreign object, and record these data. If an optical flow vector is (333.33, 166.67), then its velocity magnitude is 𝑉 ≈ 370.51 pixels / second, and the direction is 𝜃 ≈ , define an analysis method for the frequency of speed changes. Usually, time series analysis techniques (such as Fourier transform or wavelet transform) are used to identify the laws of speed changes. Set the main frequency components of the speed data extracted by Fourier transform to identify the motion change characteristics of foreign objects. Conduct frequency analysis on the recorded speed information of foreign objects, calculate its change frequency, and identify the characteristics and patterns of speed changes. If the speed of the foreign object shows periodic fluctuations within a certain period of time, identify its main frequency components through Fourier transform. According to the speed, motion direction, and speed change characteristics of the foreign object, select a method for the evolution of the comprehensive motion pattern. Cluster analysis or pattern recognition techniques can be used to extract motion patterns. Use the K-means clustering algorithm to cluster the speed and direction information of the foreign object to identify different motion patterns. Conduct comprehensive analysis on the motion characteristic data of the foreign object to construct a motion pattern model. Identify different motion patterns of the foreign object according to speed, direction, and frequency changes. Through K-means clustering analysis, classify the motion of the foreign object into different patterns such as "fast straight line" and "slow turning", and record the characteristics of these patterns. According to the motion pattern of the foreign object and the optical flow vector information, set a method for constructing a perturbed trajectory vector field. The vector field should be able to reflect the motion trajectory and intensity of the foreign object in the operation area. Set to use the optical flow vector information of each pixel point to generate a two-dimensional vector field representing the intensity and direction of motion. Combine the optical flow vector information and motion pattern data of the foreign object to construct a perturbed trajectory vector field. The vector of each pixel point should represent the motion of the foreign object at that position. Calculate the optical flow vector of each pixel point to form a complete vector field, and generate the motion trajectory within the area based on this. Record the constructed perturbed trajectory vector field in the database and display it through a visualization tool to help analyze the motion pattern of the foreign object and its impact on the surrounding environment. Generate a vector field diagram to show the motion trajectory of the foreign object in the operation area to help understand its potential impact on the operation area.

[0036] In this embodiment, step S5 includes the following steps: Conduct the evolution of the foreign object trajectory trend on the foreign object perturbed trajectory vector field to generate the foreign object trajectory trend evolution characteristics; Based on the foreign object trajectory trend evolution characteristics, conduct multi-time point trajectory prediction to generate the foreign object movement prediction trajectories at multiple time points; Obtain the current motion state information of the palletizer and conduct operation path recognition to generate the palletizer operation path; According to the foreign object movement prediction trajectory, conduct foreign object trajectory conflict prediction on the palletizer operation path to extract potential conflict path points.

[0037] In this embodiment, a suitable algorithm (such as time series analysis or trend analysis) is selected to analyze the foreign object disturbance trajectory vector field to extract the evolution characteristics of the foreign object trajectory. These characteristics should be able to reflect the changing trend of the foreign object movement. The moving average method is used to smooth the trajectory data and identify the long-term trend to reduce the influence of instantaneous fluctuations. The trajectory data of the foreign object is analyzed to calculate key characteristics such as the change in movement direction, speed change, and acceleration within each time period. Through these characteristics, the trend of the foreign object movement can be identified. Suppose that within a certain time period, the movement speed of the foreign object gradually increases, then this trend can be recorded and marked as "accelerated movement". According to the extracted trajectory trend evolution characteristics, a suitable prediction model (such as linear regression, Kalman filter, or LSTM model) is selected for multi-time point trajectory prediction. These models can predict the future movement trajectory based on historical data. The polynomial regression model is used to fit the future movement trend according to the known trajectory data. Based on the historical trajectory data and the extracted trend characteristics, the selected prediction model is used to generate the predicted trajectories of the foreign object movement at multiple time points. The predicted time points should cover the short-term and medium-term ranges in the future. If the current time is t, the trajectories at future time points such as t + 1, t + 2, t + 3, etc. are predicted to determine the possible movement paths of the foreign object. The current movement state information of the palletizer, including speed, acceleration, position, and direction, etc., is obtained through sensors and monitoring systems. This information will provide the basic data for subsequent path identification. An accelerometer and a GPS module are installed to record the movement state of the palletizer in real time. According to the obtained movement state information, the movement trajectory of the palletizer is analyzed to identify its current working path. The working path should reflect the movement trajectory of the palletizer within the working area. The path planning algorithm (such as the A* algorithm) is used to analyze the movement state to determine the working path of the palletizer. According to the predicted foreign object movement trajectory and the palletizer working path, a suitable conflict prediction algorithm (such as distance-based conflict detection or time series analysis) is selected to identify potential conflict points. A safety distance threshold is set. If the distance between the predicted foreign object trajectory and the palletizer working path is less than this threshold, it is determined as a potential conflict. The predicted trajectory of the foreign object and the working path of the palletizer are analyzed point by point to identify potential conflict path points. The time dimension needs to be considered to ensure prediction at future time points. If at a certain future time point, the predicted trajectory of the foreign object coincides with the working path of the palletizer, this point is recorded as a potential conflict point. The extracted potential conflict path points are recorded in the database and displayed through a visualization tool to help analyze the conflict risk. A conflict prediction map is generated to identify potential conflict path points, which is convenient for formulating corresponding safety measures and response strategies.

[0038] In this embodiment, step S6 includes the following steps: Perform safety passage constraint calculation on the palletizer working path to obtain the critical passage distance; Calculate the conflict probability based on the three-dimensional morphological characteristics of the foreign object for the critical passing distance, and deduce the conflict area based on the potential conflict path points to obtain the foreign object collision probability curve; Calculate the dynamic collision risk level of the foreign object collision probability curve to obtain the foreign object intrusion risk level; Make an adaptive early warning decision according to the risk level to generate an adaptive decision-making strategy for foreign object intrusion.

[0039] In this embodiment, the safety passage constraint calculation standard for the palletizer operation path is determined. The critical passage distance should be set according to the physical size of the equipment, its moving speed, the operation environment, and the three-dimensional morphological characteristics of foreign objects. This distance is usually the minimum distance required to ensure the safe operation of the equipment. The critical passage distance is set as 1.5 times the width of the palletizer plus the maximum width of the foreign object to ensure that no collision occurs during operation. Analyze the operation path of the palletizer, and combine the motion state and the three-dimensional morphological characteristics of the foreign object to evaluate the safety passage constraints in different situations. Dynamic factors such as moving speed and acceleration changes need to be considered. If the operation speed of the palletizer is 0.5 m / s and the three-dimensional morphological characteristics of the foreign object show that its width is 0.2 m, then the critical passage distance is 1.5×0.4 = 0.6 m. Based on the three-dimensional morphological characteristics of the foreign object, set the calculation method for the collision probability. The collision probability can usually be estimated through the intersection of the geometric model and the motion trajectory, and evaluated in combination with the critical passage distance. Set the calculation formula for the collision probability as: P (collision)= A(overlap) / A(total), where A(overlap) is the area of the collision region and A(total) is the total contact area. Based on the critical passage distance and the three-dimensional morphological characteristics of the foreign object, deduce the potential collision region. Use the geometric model to analyze the relative position of the foreign object and the palletizer to determine whether there is an overlapping region. Through model calculation, if the path of the foreign object overlaps with that of the palletizer, then deduce the overlapping area according to its three-dimensional morphology and calculate the corresponding collision probability. Record the calculated collision probability and the deduced collision region in the database and display them through a visualization tool to facilitate the analysis of the collision risk. Generate a collision probability heat map to show the distribution of collision probabilities in different regions and help identify high-risk regions. Set the calculation standard for the dynamic collision risk level. The assessment of the risk level should comprehensively consider the collision probability, the characteristics of the foreign object (such as material and hardness), and its potential hazards to the equipment and personnel. The risk level can be divided into low, medium, and high, and corresponding collision probability thresholds are set, such as low risk is 0 - 0.1, medium risk is 0.1 - 0.5, and high risk is above 0.5. According to the calculated foreign object collision probability curve, evaluate the current risk level. The risk level needs to be updated in real time to dynamically monitor the safety of the operation area. If the current collision probability is 0.3, then evaluate it as medium risk according to the set threshold and record the corresponding risk level information. According to the calculation result of the risk level, set the standard for the adaptive warning decision. The warning decision should include various response strategies, such as alarm triggering, operation suspension, or adjustment of the operation path, etc. Set to automatically trigger an alarm and suspend the operation of the palletizer when the risk level is high to avoid potential collisions. Based on the current risk level, generate the corresponding adaptive decision-making strategy. The strategy should be adjusted according to real-time data to ensure effective response to potential risks in different situations.If the risk level is medium, the system can recommend the operator to reduce the speed of the palletizer and closely monitor the operation area to reduce the collision risk. The generated adaptive decision-making strategy is recorded in the database and displayed through a visualization tool to help the operator understand the countermeasures. A strategy recommendation diagram is generated to show the countermeasures under different risk levels, helping the operator to make timely responses.

[0040] In this embodiment, an intrusion monitoring system based on the fusion of laser and video images is provided, which is used to execute the intrusion monitoring method based on the fusion of laser and video images as described above, and includes: A beam perturbation module, which is used to collect real-time laser scanning monitoring signals based on a multi-array laser, perform dynamic area laser occlusion perturbation mining, and construct a foreign object beam occlusion perturbation map of the operation area; A video segmentation module, which is used to obtain the monitoring video stream of the operation area according to a high-definition camera, perform adaptive cache optimization and key frame segmentation of foreign object intrusion, and extract the foreign object intrusion video frame sequence; A three-dimensional morphology module, which is used to calculate the beam occlusion density according to the beam occlusion perturbation map, and perform three-dimensional morphology analysis on the foreign object intrusion video frame sequence to obtain the three-dimensional morphological characteristics of the foreign object; A dynamic optical flow tracking module, which is used to perform dynamic optical flow tracking and vector change frequency analysis on the foreign object intrusion video frame sequence, and construct a foreign object perturbation trajectory vector field; A conflict prediction module, which is used to perform the trend evolution of the foreign object trajectory on the foreign object perturbation trajectory vector field, perform foreign object trajectory conflict prediction, and extract potential conflict path points; An intrusion decision module, which is used to perform conflict area deduction based on the potential conflict path points and the three-dimensional morphological characteristics of the foreign object, perform adaptive early warning decision-making, and generate an adaptive decision-making strategy for foreign object intrusion.

[0041] The present invention constructs a dense occlusion perturbation map through laser point cloud information, which helps to quickly lock the foreign object intrusion area in the subsequent video image; the multi-array deployment eliminates the visual blind area and improves the recognition accuracy of different incident angles and irregular objects; it provides an independent signal source based on physical interference for the entire system, enhancing the multi-source data redundancy and anti-false alarm ability of the system. Through cache optimization, full-scale analysis is avoided, and only the key frames that may be involved in intrusion are processed, reducing the computing power consumption; With the assistance of the laser map for positioning, the image analysis time is greatly shortened, and the real-time performance of video detection and positioning is improved; the cache adjustment and image enhancement technology can improve the recognition robustness under harsh conditions such as low illumination and strong backlight; after the video key frames are aligned with the laser perturbation map, the accurate definition of "when and where" the intruder is realized. Combining the laser occlusion density and the image depth feature, the three-dimensional body state, size and shape of the foreign object are accurately inverted; It can distinguish object types (such as: people, cardboard boxes, robotic arms), and adapt to differential safety responses in multiple scenarios; provide real geometric boundaries for subsequent trajectory prediction to facilitate the assessment of collision risks when objects move in space; the three-dimensional shape is used as the geometric parameter input for motion trend modeling, and the accuracy of the intrusion prediction model can be dynamically updated. Continuously restore the movement information of foreign objects at multiple time points to form a complete disturbance path; through vector frequency analysis, behavioral patterns such as rapid crossing, repeated movement, and stationary lurking can be identified; the optical flow method has a strong ability to capture fast-moving targets and can be widely applied to dynamic scenarios such as palletizing and logistics; the trajectory vector field is used as the core input of the subsequent conflict prediction algorithm to achieve path trend modeling. Based on the current trajectory trend, predict the positions of foreign objects at multiple future time points to achieve forward-looking safety control; it can compare and analyze the conflicts between the foreign object trajectory and the planned path of the palletizing machine and output potential conflict points; it can not only judge "whether there is a conflict", but also quantify the conflict time window, overlap degree and risk level; the prediction of conflict points enables the control system to reserve decision-making time such as braking, detouring, and pausing to avoid sudden interruptions. Combine the path overlap degree, motion trend and three-dimensional volume to evaluate the potential hazard level of the intruder to the operation system; automatically select response methods such as "prompt warning, action slowdown, immediate stop" according to the risk level; it can be connected to systems such as the main control PLC and robot controller to achieve automatic obstacle avoidance or behavior interruption; the hierarchical warning mechanism can avoid unnecessary control actions triggered by minor interferences and ensure production continuity.

[0042] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, it is intended to cover all changes falling within the meaning and scope of the equivalent elements of the application document within the present invention.

[0043] As described above are only the specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features invented herein.

Claims

1. An intrusion monitoring method based on the fusion of laser and video images, characterized in that Including the following steps: Step S1: Collect real-time laser scanning monitoring signals based on a multi-array laser, perform dynamic area laser shielding disturbance mining, and construct a foreign object beam shielding disturbance map for the working area; Step S2: Obtain the monitoring video stream of the working area according to a high-definition camera, perform adaptive cache optimization and key frame segmentation of foreign object intrusion, and extract the foreign object intrusion video frame sequence; Step S3: Calculate the beam shielding density according to the beam shielding disturbance map, and perform three-dimensional morphology analysis on the foreign object intrusion video frame sequence to obtain the three-dimensional morphological characteristics of the foreign object; Step S4: Perform dynamic optical flow tracking and vector change frequency analysis on the foreign object intrusion video frame sequence to construct a foreign object disturbance trajectory vector field; Step S5: Perform foreign object trajectory trend evolution on the foreign object disturbance trajectory vector field, perform foreign object trajectory conflict prediction, and extract potential conflict path points; Step S6: Based on the potential conflict path points and the three-dimensional morphological characteristics of the foreign object, perform conflict area deduction, and perform adaptive early warning decision-making to generate an adaptive decision-making strategy for foreign object intrusion.

2. The intrusion monitoring method based on laser and video image fusion according to claim 1, characterized in that, The specific steps of Step S1 are as follows: Perform continuous point cloud scanning on the working area of the palletizer based on a multi-array laser to collect real-time laser scanning monitoring signals; Perform equal-length time series division according to the real-time laser scanning monitoring signals to obtain multiple laser scanning time windows; Perform laser frequency continuity identification on the multiple laser scanning time windows to generate a laser frequency continuous steady-state curve for each time window; Perform transient mutation detection according to the laser frequency continuous steady-state curve and mark transient frequency mutation points; Calculate the frequency disturbance intensity and velocity gradient of the transient frequency mutation points, perform dynamic area laser shielding disturbance mining, and construct a foreign object beam shielding disturbance map for the working area.

3. The intrusion monitoring method based on laser and video image fusion according to claim 1, characterized in that The specific steps of Step S2 are as follows: Perform panoramic operation monitoring on the working area of the palletizer according to a high-definition camera to obtain the monitoring video stream of the working area; Perform adaptive cache optimization on the monitoring video stream of the working area to construct a delay-optimized video stream; Calculate the beam shielding time length according to the foreign object beam shielding disturbance map of the working area to obtain the intrusion behavior time curve; Locate the foreign object intrusion position according to the foreign object beam shielding disturbance map of the working area; Perform key frame segmentation of foreign object intrusion on the delay-optimized video stream according to the foreign object intrusion position and the intrusion behavior time curve, and extract the foreign object intrusion video frame sequence.

4. The intrusion monitoring method based on laser and video image fusion according to claim 3, wherein, The specific steps of performing adaptive cache optimization on the monitoring video stream of the working area to construct a delay-optimized video stream are as follows: Perform single-frame decoding of the monitoring video stream of the working area to extract the image frame sequence; Perform frame-by-frame image area segmentation on the image frame sequence to obtain multiple local areas of each image frame; Perform gray pixel calculation on the local areas to generate gray histogram information for each area; Perform adaptive gray change coefficient calculation based on the gray histogram information to generate a regional gray transformation function; Stretch the pixel gray values of the local areas according to the regional gray transformation function to generate a brightness-optimized image; Traverse all the image frame sequences to construct a brightness-optimized image frame sequence; Calculate the inter-frame delay and encoding / decoding delay of the brightness-optimized image frame sequence; Calculate the global average delay based on the inter-frame delay and the encoding / decoding delay; Based on the global average delay, perform adaptive caching optimization on the monitored video stream of the operation area to construct a delay-optimized video stream.

5. The intrusion monitoring method based on laser and video image fusion according to claim 1, characterized in that The specific steps of step S3 are as follows: Perform deep object visual detection on the foreign object intrusion video frame sequence, and mark the foreign object target nodes; Separate the bounding boxes according to the foreign object target nodes, and extract the foreign object image bounding boxes; Perform edge contour recognition on the foreign object image bounding boxes, and extract the foreign object edge contour lines; Calculate the beam occlusion density of the beam occlusion perturbation map of the foreign object in the operation area to obtain the beam occlusion density of the foreign object intrusion; Perform multi-scale spatial registration and three-dimensional shape analysis based on the beam occlusion density and the foreign object edge contour lines to obtain the three-dimensional shape characteristics of the foreign object.

6. The intrusion monitoring method based on laser and video image fusion according to claim 1, wherein, The specific steps of step S4 are as follows: Perform dynamic optical flow tracking on the foreign object intrusion video frame sequence to identify the pixel point movement trajectories of the frame sequence; Perform optical flow vector calculation on the pixel point movement trajectories to obtain the foreign object optical flow vector information; Calculate the magnitude and direction of the foreign object optical flow vector information to obtain the speed and movement direction of the foreign object; Perform vector change frequency analysis in the time dimension on the foreign object optical flow vector information to obtain the foreign object movement speed change characteristics; Perform comprehensive motion mode evolution on the foreign object movement speed change characteristics, speed and movement direction to construct a foreign object perturbation trajectory vector field.

7. The intrusion monitoring method based on laser and video image fusion according to claim 1, characterized in that The specific steps of step S5 are as follows: Perform foreign object trajectory trend evolution on the foreign object perturbation trajectory vector field to generate foreign object trajectory trend evolution characteristics; Perform multi-time point trajectory prediction based on the foreign object trajectory trend evolution characteristics to generate foreign object movement prediction trajectories at multiple time points; Obtain the current motion state information of the palletizer, and perform operation path recognition to generate the palletizer operation path; Perform foreign object trajectory conflict prediction on the palletizer operation path according to the foreign object movement prediction trajectories, and extract potential conflict path points.

8. The intrusion monitoring method based on laser and video image fusion according to claim 1, characterized in that The specific steps of step S6 are as follows: Perform safety passage constraint calculation on the palletizer operation path to obtain the critical passage distance; Calculate the conflict probability based on the three-dimensional shape characteristics of the foreign object for the critical passage distance, and perform conflict area deduction based on the potential conflict path points to obtain the foreign object collision probability curve; Perform dynamic collision risk level calculation on the foreign object collision probability curve to obtain the foreign object intrusion risk level; Perform adaptive warning decision according to the risk level to generate a foreign object intrusion adaptive decision strategy.

9. An intrusion monitoring system based on the fusion of laser and video images, characterized in that, For executing the intrusion object monitoring method based on laser and video image fusion as described in claim 1, including: A beam perturbation module, configured to collect real-time laser scanning monitoring signals based on a multi-array laser, and perform dynamic area laser occlusion perturbation mining to construct a foreign object beam occlusion perturbation map of the operation area; A video segmentation module, configured to obtain a monitored video stream of the operation area according to a high-definition camera, and perform adaptive caching optimization and key frame segmentation of foreign object intrusion to extract a foreign object intrusion video frame sequence; A three-dimensional shape module, configured to calculate the beam occlusion density according to the beam occlusion perturbation map, and perform three-dimensional shape analysis on the foreign object intrusion video frame sequence to obtain the three-dimensional shape characteristics of the foreign object; The dynamic optical flow tracking module is used to perform dynamic optical flow tracking and vector change frequency analysis on the foreign object intrusion video frame sequence, and construct a foreign object disturbance trajectory vector field; The conflict prediction module is used to evolve the foreign object trajectory trend of the foreign object disturbance trajectory vector field, predict foreign object trajectory conflicts, and extract potential conflict path points; The intrusion decision module is used to deduce the conflict area based on the potential conflict path points and the three-dimensional morphological characteristics of the foreign object, and make an adaptive early warning decision to generate an adaptive decision-making strategy for foreign object intrusion.

Images