Visual identification bin handling control method and system

Abstract

The application relates to the technical field of industrial automation control, and discloses a visual identification material box carrying control method and system. Image data containing material box positions and postures is acquired in real time by visual acquisition equipment arranged in a carrying area, compared with a pre-stored standard material box feature template, combined with the ambient light intensity, ground flatness and material box size parameters of the current carrying scene, and reference carrying parameters are determined. Difference analysis is carried out on the real-time image data and the reference parameters, abnormal areas deviating from the tolerance range in position or posture are identified and marked as carrying optimization areas, carrying adjustment strategies containing specific methods and adjustment values of carrying direction correction or posture correction are generated according to the image data change trend and the deviation degree of the areas, and finally the adjustment strategies are executed by the actuators arranged in the carrying equipment to complete accurate carrying control. The application can adapt to different environments and improve the accuracy and efficiency of material box carrying.

Description

Technical Field

[0001] This invention relates to the field of industrial automation control technology, specifically to a visual recognition-based bin handling control method and system. Background Technology

[0002] In modern industrial production, bin handling is a crucial component of logistics warehousing and production line transfer. Traditional bin handling control methods often rely on preset program parameters or manual operation, making them difficult to adapt to complex and ever-changing real-world operating environments. For example, fluctuations in ambient light intensity can cause deviations in image data acquired by visual recognition equipment during bin handling; differences in ground flatness may cause bins to shift position or change orientation during transport; and bins of different sizes require different handling parameters.

[0003] Existing technical solutions typically cannot perceive these environmental changes and bin statuses in real time and with high accuracy, making it difficult to dynamically adjust handling strategies. This leads to problems such as positional deviations and abnormal postures during bin handling, affecting handling efficiency and accuracy, and potentially causing bin damage or production interruptions. Furthermore, traditional methods often suffer from lag in anomaly detection, failing to promptly identify deviations in bin position or posture, and struggling to generate reasonable adjustment strategies based on the degree of deviation, thus limiting the accuracy and reliability of handling control.

[0004] With the continuous improvement of industrial automation, the requirements for intelligent and precise control of bin handling are increasing. There is an urgent need for a method that can combine visual recognition technology to adapt to environmental changes in real time and precisely control bin handling. Summary of the Invention

[0005] The purpose of this invention is to provide a visual recognition-based bin handling control method and system to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides a visual recognition-based bin handling control method, the method comprising:

[0007] During the material box handling operation, image data including the position and posture of the material box are acquired in real time by vision acquisition devices deployed in the handling area;

[0008] The real-time acquired image data is compared with the pre-stored standard bin feature template. Combined with the ambient light intensity, ground flatness and bin size parameters of the current handling scenario, the baseline handling parameters of the bin in the current scenario are determined.

[0009] The difference analysis between the real-time acquired image data and the baseline transport parameters is performed to identify abnormal areas where the position or orientation deviates beyond the tolerance range, and these abnormal areas are marked as transport optimization areas that need to be adjusted.

[0010] For the marked transport optimization area, a transport adjustment strategy is generated based on the changing trend and deviation of the image data. The adjustment strategy includes the specific method and adjustment value of transport direction correction or posture correction.

[0011] By executing the handling adjustment strategy through the actuators installed on the handling equipment, the precise handling control operation of the material box is completed.

[0012] Preferably, image data including the position and orientation of the material bins is acquired in real time using vision acquisition devices deployed in the handling area. The specific steps are as follows:

[0013] Industrial cameras are installed at the entrance of the transport channel, the operation point of the transport equipment, and the edge of the material bin storage area. Each camera synchronously captures images at fixed time intervals.

[0014] After the image information captured by each camera is converted into a digital signal, it is transmitted to the main control processing unit through the communication module. The main control processing unit restores the received digital signal into image data and stores it as a continuous image sequence according to the acquisition time order.

[0015] Preferably, the real-time acquired image data is compared with the pre-stored standard bin feature template, and combined with the ambient light intensity, ground flatness, and bin size parameters of the current handling scenario, the baseline handling parameters of the bin in the current scenario are determined. The specific steps are as follows:

[0016] Retrieve historical scene data from the standard bin feature template library that are the same size as the current bin, with ambient light intensity difference not exceeding the set range, and ground flatness difference not exceeding the set range;

[0017] Statistical processing is performed on the retrieved historical scene data to calculate the median value and fluctuation range of the material box handling position and posture in the historical scene data, and the median value is used as the initial benchmark handling parameter.

[0018] The initial reference handling parameters are corrected based on the current ambient temperature and humidity, the reflectivity of the material bin surface, and the initial position parameters of the handling equipment to obtain the final reference handling parameters.

[0019] Preferably, a difference analysis is performed on the real-time acquired image data and the baseline transport parameters to identify abnormal areas where the position or orientation deviates beyond the tolerance range, and these abnormal areas are marked as transport optimization areas that need adjustment. The specific steps are as follows:

[0020] The real-time acquired continuous image sequence is compared frame by frame with the baseline handling parameters to calculate the deviation of the position and orientation of the hopper in each frame.

[0021] Set the maximum tolerance range for deviation values. If the deviation value of a certain frame exceeds the maximum tolerance range, then the frame is marked as an abnormal frame.

[0022] The sliding window technique is used to process the marked abnormal frames, and the image intervals in which consecutive abnormal frames appear are merged into a continuous abnormal region. This continuous abnormal region is the transport optimization region that needs to be adjusted.

[0023] Preferably, for the marked transport optimization area, a transport adjustment strategy is generated based on the changing trend and deviation of the image data. The adjustment strategy includes the specific methods and adjustment values ​​for transport direction correction or attitude correction. The specific steps are as follows:

[0024] Analyze the trend of continuous image sequences within the transport optimization area. If the position coordinates show a continuous leftward shift, the transport direction correction method is determined to be rightward shift; if the attitude angle shows a continuous forward tilt, the attitude correction method is determined to be backward tilt.

[0025] Calculate the average deviation value within the material handling optimization area, and divide the degree of deviation into three levels: slight, moderate and severe based on the magnitude of the average value. Each level corresponds to a preset adjustment value, and finally generate a material handling adjustment strategy that includes correction methods and adjustment values.

[0026] Preferably, the material bin is precisely controlled by executing a handling adjustment strategy through an actuator installed on the handling equipment. The specific steps are as follows:

[0027] The actuator of the handling equipment includes a rotatable robotic arm and a gripping device connected to the robotic arm. The driving component of the actuator is a servo drive motor that is matched with the robotic arm.

[0028] Based on the correction method and adjustment value in the handling adjustment strategy, a control signal is sent to the servo drive motor. The servo drive motor drives the robotic arm to rotate or extend, which in turn moves the gripping device, thereby changing the handling position or posture of the hopper and realizing the handling control of the hopper.

[0029] Preferably, a maximum tolerance range for deviation values ​​is set. If the deviation value of a certain frame exceeds the maximum tolerance range, the frame is marked as an abnormal frame. The specific steps are as follows:

[0030] By analyzing the fluctuation characteristics of historical scenario data in the standard bin feature template library, the upper limit and lower limit of normal fluctuation of the deviation value are determined. The range between the upper limit and the lower limit of normal fluctuation is the maximum tolerance range of the deviation value.

[0031] The deviation value of each frame of image acquired in real time is determined. If the deviation value is greater than the upper limit of normal fluctuation or less than the lower limit of normal fluctuation, the frame of image is marked as an abnormal frame.

[0032] Preferably, the average deviation value within the optimized handling area is calculated, and the degree of deviation is divided into three levels: slight, moderate, and severe based on the magnitude of the average value. The specific steps are as follows:

[0033] Calculate the sum of the deviation values ​​of all abnormal frames within the transport optimization area, divide by the number of abnormal frames, and obtain the average deviation value;

[0034] If the deviation from the average is less than the set slight deviation threshold, the deviation is classified as slight; if the deviation from the average is greater than or equal to the slight deviation threshold but less than the set serious deviation threshold, the deviation is classified as moderate; if the deviation from the average is greater than or equal to the serious deviation threshold, the deviation is classified as serious.

[0035] Preferably, a control signal is sent to the servo drive motor according to the correction method and adjustment value in the handling adjustment strategy. The specific steps are as follows:

[0036] If the correction method is to move the transport direction to the right, the control signal is to increase the output torque of the servo drive motor, so that the robotic arm rotates to the right and the gripping device moves to the right away from the current position.

[0037] If the correction method is a backward tilt correction posture, the control signal is to reduce the output torque of the servo drive motor, so that the robotic arm extends and retracts backward, and the gripping device drives the material box to tilt backward.

[0038] The adjustment value is achieved by controlling the change in the output torque of the servo drive motor, and the change in torque is directly proportional to the adjustment value.

[0039] Preferably, this method further includes a vision-based bin handling control system, which is used to implement the above-described vision-based bin handling control method, the system comprising:

[0040] Visual acquisition equipment is deployed in the handling area to acquire image data containing the position and posture of the bins in real time during the bin handling operation;

[0041] The parameter determination module is used to compare the image data acquired in real time by the vision acquisition device with the pre-stored standard bin feature template, and combine the ambient light intensity, ground flatness and bin size parameters of the current handling scenario to determine the reference handling parameters of the bin in the current scenario.

[0042] An anomaly identification module is used to perform difference analysis between the image data acquired in real time by the visual acquisition device and the benchmark handling parameters determined by the parameter determination module, identify abnormal areas where the position or posture deviates beyond the tolerance range, and mark the abnormal areas as handling optimization areas that need to be adjusted.

[0043] The strategy generation module is used to generate a transport adjustment strategy for the transport optimization area marked by the anomaly recognition module, based on the changing trend and deviation of the image data. The adjustment strategy includes the specific method and adjustment value of transport direction correction or posture correction.

[0044] The execution module, located on the handling equipment, is used to execute the handling adjustment strategy generated by the strategy generation module to complete the precise handling control operation of the material box.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] This invention, by deploying visual acquisition devices in the handling area, can acquire real-time image data including the position and posture of the material bins. Compared to traditional methods that rely on manual experience or fixed procedures for material bin handling, this improves the real-time perception of the material bin status, laying a solid foundation for subsequent precise handling control. The real-time image data is compared with pre-stored standard material bin feature templates, and benchmark handling parameters are determined by combining ambient light intensity, ground flatness, and material bin size parameters. This method fully considers various influencing factors in the actual working environment, making the determined benchmark parameters more suitable for the current scenario and effectively improving the adaptability and accuracy of the handling parameters.

[0047] By performing difference analysis between real-time image data and baseline parameters, abnormal areas where position or posture deviations exceed tolerance limits can be identified in a timely manner, preventing the accumulation and expansion of anomalies and providing a basis for timely adjustments to the handling strategy. For marked handling optimization areas, handling adjustment strategies are generated based on the changing trends and deviation degrees of the image data, achieving personalized and precise adjustment strategies and ensuring that the adjustment methods and values ​​accurately address different degrees of deviation. The handling adjustment strategy is executed by the actuators installed on the handling equipment. Utilizing the precise control of components such as servo drive motors, accurate correction of the material box's handling direction and posture can be achieved, thereby significantly improving the accuracy of material box handling and reducing deviations and errors during the handling process.

[0048] The entire method achieves fully automated control from image acquisition, parameter determination, anomaly recognition to strategy generation and execution. It not only improves the efficiency of bin handling but also reduces the cost and error of manual intervention, enhances the stability and reliability of handling operations, and can better meet the high-efficiency and precise requirements of bin handling in modern industrial production. Attached Figure Description

[0049] Figure 1 This is a schematic diagram illustrating the working principle of the visual recognition bin handling control method described in this invention.

[0050] Figure 2 Flowchart for determining the baseline transport parameters;

[0051] Figure 3 A flowchart for actuator control;

[0052] Figure 4 This is a flowchart for determining abnormal frames. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Please see Figures 1-4 This invention provides a visual recognition-based bin handling control method, the specific implementation steps of which are as follows:

[0055] During the material box handling operation, visual acquisition devices deployed in the handling area acquire image data containing the position and posture of the material boxes in real time. Industrial cameras are set up at the entrance of the handling channel, the working point of the handling equipment, and the edge of the material box storage area. Each camera synchronously acquires images at fixed time intervals. After the image information acquired by each camera is converted into digital signals, it is transmitted to the main control processing unit through the communication module. The main control processing unit restores the received digital signals into image data and stores them as a continuous image sequence according to the acquisition time order.

[0056] The real-time acquired image data is compared with pre-stored standard bin feature templates. Combined with the ambient light intensity, ground flatness, and bin size parameters of the current handling scenario, the baseline handling parameters for the bin in the current scenario are determined. Historical scene data with the same bin size, ambient light intensity difference not exceeding a set range, and ground flatness difference not exceeding a set range are retrieved from the standard bin feature template library. Statistical processing is performed on the retrieved historical scene data to calculate the median value and fluctuation range of the bin's handling position and posture. The median value is used as the initial baseline handling parameters. These initial baseline handling parameters are then corrected based on the current ambient temperature and humidity, bin surface reflectivity, and the initial position parameters of the handling equipment to obtain the final baseline handling parameters.

[0057] A difference analysis is performed on the real-time acquired image data and the baseline handling parameters to identify abnormal areas where the position or orientation deviation exceeds the tolerance range. These abnormal areas are marked as handling optimization areas requiring adjustment. The real-time acquired continuous image sequence is compared frame-by-frame with the baseline handling parameters. The deviation value of the bin's position and orientation in each frame is calculated. By analyzing the fluctuation characteristics of historical scene data in the standard bin feature template library, the upper and lower limits of normal fluctuation for the deviation value are determined. The interval between the upper and lower limits is the maximum tolerance range for the deviation value. The deviation value of each real-time acquired image frame is judged. If the deviation value is greater than the upper limit or less than the lower limit, the image frame is marked as an abnormal frame. A sliding window technique is used to process the marked abnormal frames, merging image intervals with consecutive abnormal frames into continuous abnormal regions. These continuous abnormal regions are the handling optimization areas requiring adjustment.

[0058] For the marked transport optimization area, a transport adjustment strategy is generated based on the changing trend and deviation degree of the image data. The adjustment strategy includes the specific method and adjustment value of transport direction correction or attitude correction. The trend of continuous image sequences within the transport optimization area is analyzed. If the position coordinates show a continuous leftward shift, the transport direction correction method is determined to be rightward shift; if the attitude angle shows a continuous forward tilt, the attitude correction method is determined to be backward tilt. The sum of the deviation values ​​of all abnormal frames within the transport optimization area is calculated and divided by the number of abnormal frames to obtain the average deviation. If the average deviation is less than a set slight deviation threshold, the deviation degree is classified as slight; if the average deviation is greater than or equal to the slight deviation threshold but less than a set severe deviation threshold, the deviation degree is classified as moderate; if the average deviation is greater than or equal to the severe deviation threshold, the deviation degree is classified as severe. Based on the magnitude of the average value, the deviation degree is divided into three levels: slight, moderate, and severe. Each level corresponds to a preset adjustment value, ultimately generating a transport adjustment strategy that includes the correction method and adjustment value.

[0059] By setting the actuator of the handling device to execute the handling adjustment strategy, the precise handling control operation of the bin is completed. The actuator of the handling device includes a rotatable robotic arm and a gripping device connected to the robotic arm. The driving component of the actuator is a servo drive motor supporting the robotic arm. According to the correction method and adjustment value in the handling adjustment strategy, a control signal is sent to the servo drive motor. If the correction method is to shift the handling direction to the right, the control signal is to increase the output torque of the servo drive motor, causing the robotic arm to rotate to the right and the gripping device to move to the right away from the current position. If the correction method is to correct the posture by tilting backward, the control signal is to decrease the output torque of the servo drive motor, causing the robotic arm to extend backward and the gripping device to tilt the bin backward. The adjustment value is achieved by controlling the change range of the output torque of the servo drive motor, and the torque change range is in a proportional relationship with the adjustment value. The servo drive motor drives the robotic arm to rotate or extend,带动 the gripping device to move, thereby changing the handling position or posture of the bin and实现 the handling control of the bin.

[0060] Example 1:

[0061] During the bin handling operation, it is necessary to obtain the image data containing the position and posture of the bin in real time through the visual acquisition device deployed in the handling area.

[0062] The deployment positions of the visual acquisition device cover the entrance of the handling passage, the operation point of the handling device, and the edge of the bin storage area. Industrial cameras are respectively set at these positions. Each industrial camera performs synchronous image acquisition at a fixed time interval, and this fixed time interval can be set according to the actual handling operation requirements and scene characteristics, such as set to acquire once per second or once every half second, etc., to ensure that the image information of the bin during handling can be obtained in real time and continuously.

[0063] After each industrial camera completes image acquisition, it will process the acquired image information and convert it into a digital signal. This conversion process is achieved through the relevant circuits and chips inside the camera, and its purpose is to facilitate subsequent signal transmission and data processing. The converted digital signal is transmitted to the main control processing unit through the communication module. The communication module can be a wired communication module, such as an Ethernet module, or a wireless communication module, such as a Wi-Fi module or a Bluetooth module, etc. The specific communication method can be determined according to the actual environment and communication requirements of the handling area to ensure that the digital signal can be transmitted stably and reliably.

[0064] After receiving the digital signals from the communication module, the main control processing unit processes these signals to convert them into image data. The main control processing unit typically has corresponding signal processing algorithms and programs that can decode and filter the digital signals to recover the original image information. The recovered image data is stored in chronological order of acquisition, forming a continuous image sequence. During storage, a timestamp is added to each frame for subsequent time-series analysis and processing. The storage medium can be the internal storage chip of the main control processing unit or an external storage device such as a hard drive or solid-state drive to ensure sufficient storage capacity for the image data.

[0065] By installing industrial cameras at the entrance of the transport channel, images can be captured as the bins enter the transport area, obtaining their initial position and orientation information. Installing industrial cameras at the operating points of the transport equipment allows for real-time monitoring of the interaction between the equipment and the bins, such as changes in the bin's position and orientation during gripping and transport operations. Finally, installing industrial cameras at the edge of the bin storage area allows for the acquisition of the bin's position and orientation information as it is about to be stored, ensuring accurate placement.

[0066] Each industrial camera synchronously acquires images at fixed time intervals, ensuring good temporal consistency and continuity of the acquired image data. This makes subsequent analysis of the material box's movement trajectory and attitude changes more accurate. For example, when the material box shifts position or changes attitude during handling, the process and trend of these changes can be clearly observed through a continuous sequence of images.

[0067] During digital signal transmission, the communication module encodes and modulates the signal to improve its anti-interference capability and transmission efficiency. When the main control processing unit restores the received digital signal, it performs noise filtering and image enhancement to improve the quality of the image data and make the position and orientation of the material bin in the image clearer and more identifiable.

[0068] The stored continuous image sequence provides rich image information for subsequent bin handling control. For example, when determining baseline handling parameters, the actual position and orientation of the bin in the current scenario can be obtained by analyzing this image data; when performing difference analysis, the real-time acquired image data can be compared with the baseline handling parameters to identify abnormal areas; when generating handling adjustment strategies, the specific adjustment method and adjustment value can also be determined based on the changing trend and deviation of the image data.

[0069] By setting up industrial cameras at specific locations and synchronously acquiring images at fixed time intervals, converting the image information into digital signals and transmitting them to the main control processing unit, which then restores and stores them as a continuous image sequence, this series of operations enables real-time acquisition of image data containing the position and attitude of the material box. This provides accurate and reliable image data support for subsequent material box handling control, ensuring the smooth implementation of the material box handling control method.

[0070] Example 2:

[0071] When determining the baseline handling parameters of the material bin in the current scenario, it is necessary to compare the real-time acquired image data with the pre-stored standard material bin feature template, and combine this with the ambient light intensity, ground flatness, and material bin size parameters of the current handling scenario. The specific implementation method is as follows:

[0072] Retrieve historical scene data from the standard bin feature template library. This library stores a large amount of bin handling data under various conditions. Retrieval requires specific conditions to be met: the bin size must be the same as the current bin size, and the differences in ambient light intensity and ground flatness must not exceed set ranges. These set ranges are predetermined based on actual application scenarios and experience. For example, the difference in ambient light intensity can be set to no more than ±100 lux, and the difference in ground flatness can be set to no more than ±5 mm / m. Specific values ​​can be adjusted according to actual conditions.

[0073] After retrieving historical scene data that meets the criteria, statistical processing is required. The purpose of this statistical processing is to extract representative information from a large amount of historical scene data to determine the initial baseline handling parameters. Specifically, it is necessary to calculate the median and fluctuation range of the material box handling position and attitude in the historical scene data. The median can be calculated as an arithmetic mean, which involves adding the handling position coordinates (such as x, y, z coordinates) and attitude angles (such as pitch, yaw, and roll angles) of all historical scene data, and then dividing by the number of data points to obtain the median value for each dimension. The fluctuation range can be determined by calculating the standard deviation or range. For example, with the median value as the center, extending to the left and right by a certain multiple of the standard deviation forms an interval, which is the fluctuation range. Through this statistical processing, an initial baseline handling parameter that can represent the overall trend of the historical scene data can be obtained.

[0074] After obtaining the initial baseline handling parameters, they need to be corrected based on the current ambient temperature and humidity, the reflectivity of the material bin surface, and the initial position parameters of the handling equipment to obtain the final baseline handling parameters. The current ambient temperature and humidity affect the operating status of the handling equipment and the physical characteristics of the material bin. For example, temperature changes may cause thermal expansion and contraction of mechanical parts, thus affecting the accuracy of the handling equipment; humidity changes may alter the friction of the material bin surface, thus affecting the handling posture of the material bin. Different reflectivity of the material bin surface will lead to differences in the image quality acquired by the visual acquisition equipment. A material bin surface with high reflectivity may produce glare, affecting the accurate identification of the material bin's position and posture. Therefore, the initial baseline parameters need to be adjusted according to the reflectivity. The initial position parameters of the handling equipment determine the starting position of the handling equipment when the handling operation begins. Different initial positions may require different handling parameters to ensure the accuracy and stability of the handling.

[0075] When making corrections based on temperature and humidity, a correspondence table or mathematical model can be established between temperature / humidity and the correction amounts for handling parameters. For example, when the temperature rises, the length of the robotic arm may increase slightly, leading to a deviation in the handling position. In this case, the coordinates of the handling position need to be adjusted accordingly based on the temperature change. Similarly, when the humidity increases, the friction between the hopper and the gripping device may decrease. To ensure gripping stability, the reference parameters of the attitude angle need to be corrected.

[0076] The impact of the reflectivity of the bin surface can be addressed through experiments or empirical observation to determine the error range of image recognition under different reflectivity levels, and the initial baseline handling parameters can be corrected accordingly. For example, when the reflectivity of the bin surface is high, the edge of the bin in the image acquired by the vision acquisition device may become blurred, causing a deviation between the identified position and orientation and the actual situation. In this case, it is necessary to appropriately adjust the baseline parameters of the handling position and orientation based on the reflectivity to compensate for the image recognition error.

[0077] Correcting the initial position parameters of the handling equipment requires considering the movement trajectory and attitude changes of the equipment from the initial position to the target position. Different initial positions may require different handling directions and attitude adjustment methods. Therefore, it is necessary to modify the handling direction and attitude correction parameters in the initial reference handling parameters according to the initial position parameters to ensure that the handling equipment can move accurately from the initial position to the target position and maintain the correct attitude.

[0078] Through the above steps, historical scenario data that meets the conditions is first retrieved and statistically processed to obtain initial baseline handling parameters. These parameters are then corrected by considering the current environmental temperature and humidity, the reflectivity of the material bin surface, and the initial position parameters of the handling equipment, ultimately yielding baseline handling parameters suitable for the current scenario. This process fully utilizes the reference value of historical scenario data while also considering the specific characteristics of the current scenario, ensuring that the determined baseline handling parameters more accurately reflect the actual needs of material bin handling. This provides a reliable basis for subsequent difference analysis, anomaly identification, and the generation of handling adjustment strategies.

[0079] Example 3:

[0080] When performing difference analysis between real-time acquired image data and baseline transport parameters to identify abnormal areas where the position or orientation deviates beyond the tolerance range, and marking these abnormal areas as transport optimization areas requiring adjustment, the specific implementation method is as follows:

[0081] The real-time acquired continuous image sequence is compared frame-by-frame with the baseline handling parameters. The real-time acquired continuous image sequence is synchronously acquired and stored by the vision acquisition device at fixed time intervals. Each frame contains the position and attitude information of the bin at a specific moment. The baseline handling parameters are determined through the previous steps and represent the ideal handling position and attitude of the bin in the current scene. During frame-by-frame comparison, the position coordinates (such as x, y, and z axis coordinates) and attitude angles (such as pitch angle, yaw angle, and roll angle) of the bin in each frame need to be compared with the corresponding values ​​in the baseline handling parameters to calculate the deviation value of the bin's position and attitude in each frame. For example, if the x-axis coordinate of the bin in a certain frame is 1050mm, while the ideal x-axis coordinate in the baseline handling parameters is 1000mm, then the deviation value of the frame in the x-axis direction is 50mm; if the pitch angle of the bin in a certain frame is 5°, while the pitch angle in the baseline parameters should be 0°, then the pitch angle deviation value of the frame is 5°.

[0082] It is necessary to set a maximum tolerance range for deviation values. This range is determined by analyzing the fluctuation characteristics of historical scenario data in the standard bin feature template library. The standard bin feature template library stores a large amount of historical scenario data on bin handling under different conditions, including various possible fluctuations in bin position and orientation. By performing statistical analysis on this historical scenario data, such as calculating the average and standard deviation of deviation values ​​for each dimension, the upper and lower limits of normal fluctuation can be determined. The interval between the upper and lower limits of normal fluctuation is the maximum tolerance range for deviation values. For example, if analysis of historical scenario data reveals that a deviation of a certain orientation angle within ±3° is common, then the maximum tolerance range for that orientation angle can be set to -3° to +3°.

[0083] After determining the maximum tolerance range, the deviation value of each frame of the image acquired in real time needs to be judged. For the deviation value of each dimension in each frame of the image (such as the deviation values ​​of position along the x, y, and z axes, and the deviation values ​​of each angle of attitude), it is determined whether it exceeds the corresponding maximum tolerance range. If the deviation value of any dimension in a frame of the image is greater than the upper limit of normal fluctuation or less than the lower limit of normal fluctuation, then the frame of the image is marked as an abnormal frame. For example, if the x-axis deviation value of a frame of the image is +40mm, while the upper limit of normal fluctuation for that dimension is +30mm, then the frame of the image will be marked as an abnormal frame; or if the pitch angle deviation value of a frame of the image is -4°, while the lower limit of normal fluctuation is -3°, it will also be marked as an abnormal frame.

[0084] After identifying anomalous frames, a sliding window technique is used to process them. The sliding window technique is a method for handling consecutive anomalies in time-series data. It detects consecutive anomalous frame intervals by sliding this window across the image sequence, setting a window size (i.e., the number of consecutive frames). Specifically, a sliding window length is set, for example, 5 frames. Then, starting from the first frame, the number of anomalous frames within each window is checked sequentially. If a certain number of consecutive anomalous frames appear within a window (e.g., all frames within the window are anomalous), the image interval corresponding to that window is considered a consecutive anomalous region. By moving the sliding window, all image intervals with consecutive anomalous frames are merged into a consecutive anomalous region, which is the area requiring adjustment and optimization. For example, in an image sequence where frames 10 to 15 are six consecutive anomalous frames, when the sliding window length is set to 5 frames, both the windows for frames 10-14 and 11-15 will detect consecutive anomalous frames, thus merging frames 10-15 into a single consecutive anomalous region.

[0085] By comparing and calculating deviation values ​​frame by frame, subtle changes in the position and posture of the hopper in each frame can be accurately captured. The maximum tolerance range is determined based on the fluctuation characteristics of historical scene data, ensuring a scientific basis for setting the tolerance range and adapting to normal fluctuations in actual handling scenarios. The determination of deviation values ​​for each frame ensures accurate marking of abnormal frames. The application of sliding window technology merges scattered abnormal frames into continuous abnormal regions, thereby accurately locating the specific areas requiring handling adjustments. This series of steps is interconnected; from data comparison to anomaly detection and region location, each step strictly follows established methods and procedures, ensuring accurate identification of the handling optimization areas requiring adjustment and providing a clear target area for subsequent handling adjustment strategy generation.

[0086] Example 4:

[0087] When generating a transport adjustment strategy for the marked transport optimization area, it is necessary to determine the specific adjustment method and value based on the changing trend and deviation of the image data. The specific implementation method is as follows:

[0088] This study analyzes the variation trend of a continuous image sequence within the transport optimization region. The transport optimization region is an image interval determined by merging consecutive abnormal frames using a sliding window technique. It includes the continuous change process of the bin's position or orientation deviating from the baseline parameters. Taking position coordinates as an example, assuming that within a certain transport optimization region, the bin's x-axis coordinates for 10 consecutive frames are 1020mm, 1025mm, 1030mm, 1035mm, 1040mm, 1045mm, 1050mm, 1055mm, 1060mm, and 1065mm, while the ideal x-axis coordinate in the baseline transport parameters is 1000mm. By observing this data, it can be found that the x-axis coordinate continuously increases from 1020mm to 1065mm, showing a clear trend of continuous leftward shift (assuming the positive direction of the x-axis in the coordinate system is left). At this point, it can be determined that the correction method for the transport direction is to shift to the right to offset the continuous leftward deviation.

[0089] Taking attitude angle as an example, if the pitch angle of the hopper in 8 consecutive frames of images in a certain handling optimization area is 5°, 6°, 7°, 8°, 9°, 10°, 11° and 12° respectively, while the pitch angle in the reference parameter should be 0°, it can be seen that the attitude angle shows a continuous forward tilting trend. At this time, the attitude correction method should be determined to be backward tilting. The hopper is tilted backward by the movement of the robotic arm to restore it to the ideal attitude.

[0090] Calculate the average deviation within the optimized transport region. Specifically, first sum the deviation values ​​of all abnormal frames within the region, then divide by the number of abnormal frames. For example, if an optimized transport region contains 15 abnormal frames with x-axis deviations of 20mm, 22mm, 25mm, 28mm, 30mm, 32mm, 35mm, 38mm, 40mm, 42mm, 45mm, 48mm, 50mm, 52mm, and 55mm respectively, summing these values ​​gives a total of 537mm. Dividing this by 15 yields an average deviation of 35.8mm.

[0091] The degree of deviation is categorized into three levels: slight, moderate, and severe, based on the magnitude of the deviation from the average value. The threshold values ​​for each level need to be preset according to the actual application scenario and the accuracy requirements of the material handling. Assuming a slight deviation threshold is set at 20mm and a severe deviation threshold at 40mm: if the average deviation of a certain area is 15mm, less than 20mm, the deviation is classified as slight; if the average deviation is 30mm, between 20mm and 40mm, it is classified as moderate; and if the average deviation is 45mm, greater than 40mm, it is classified as severe.

[0092] Different degrees of deviation correspond to different preset adjustment values. For example, when the deviation is slight, the preset adjustment value is 10mm; for moderate deviation, it is 30mm; and for severe deviation, it is 50mm. Referring to the previous example, if a certain area deviates from the average value by 35.8mm, which is considered a moderate deviation, the corresponding adjustment value is 30mm. Since the deviation trend is a continuous leftward shift along the x-axis, the correction method is a rightward shift, resulting in a final handling adjustment strategy of "shift 30mm to the right".

[0093] The average pitch angle deviation within a certain transport optimization area is 8°. If the preset threshold for slight deviation is 5° and the threshold for severe deviation is 10°, then 8° is considered a moderate deviation. Assuming the attitude correction adjustment value corresponding to moderate deviation is 6°, and since the attitude shows a continuous forward tilting trend, the correction method is backward tilting, then the adjustment strategy is "backward tilting 6°".

[0094] When generating adjustment strategies, deviations in position and attitude must be considered comprehensively. If a region exhibits both leftward shift in position and forward tilt in attitude, the trends and degrees of deviation for each must be analyzed separately. For example, if the average deviation of the x-axis position is 30mm (moderate deviation, corresponding to a 30mm rightward shift) and the average deviation of the pitch angle is 7° (moderate deviation, corresponding to a 7° backward tilt), then the final adjustment strategy must include both position correction and attitude correction, i.e., "30mm rightward shift, 7° backward tilt".

[0095] The generation process of adjustment strategies must be strictly based on the actual changing trends and deviation values ​​of the image data. By analyzing continuous image sequences frame by frame, the direction and magnitude of the deviation can be accurately captured, while the hierarchical classification of the deviation degree and the preset adjustment values ​​ensure the standardization and operability of the strategy. For example, when a material bin gradually shifts position during handling due to slight unevenness of the ground, the continuous image sequence will reflect a slow deviation trend. By calculating the average deviation and classifying it into levels, a small-amplitude adjustment strategy matching this trend can be generated. If the deviation increases rapidly due to a large initial positional deviation of the handling equipment or a sudden change in ambient lighting, the deviation value of the continuous images will quickly exceed the tolerance range. In this case, the generated adjustment strategy needs to correspond to a larger adjustment value to quickly correct the deviation.

[0096] The correction method in the adjustment strategy must strictly correspond to the deviation trend. If the position coordinates show a continuous rightward shift (assuming the positive x-axis direction in the coordinate system is to the right), the correction method should be leftward; if the posture angle shows a continuous backward tilting trend, the correction method should be forward tilting. This correspondence is based on the kinematic principles of bin handling, ensuring that the adjustment action can directly counteract the deviation. For example, when the gripping device causes the bin to shift to the right, rotating the robotic arm to the left will straighten the bin; when the bin tilts backward, causing a shift in the center of gravity, extending and retracting the robotic arm forward can adjust the bin's posture.

[0097] Example 5:

[0098] When a handling adjustment strategy is executed through an actuator installed on a handling device, the mechanical structure and drive components of the actuator are used to translate the generated adjustment strategy into actual mechanical actions, thereby achieving precise handling control of the hopper. The specific implementation method is as follows:

[0099] The actuator of a material handling device includes a rotatable robotic arm and a gripping device connected to the robotic arm. The drive component of the actuator is a servo drive motor that is matched with the robotic arm. Taking a common six-axis industrial robotic arm as an example, the robotic arm can rotate and extend in three-dimensional space. The gripping device is usually a pneumatic or electric clamp that can stably hold the material box. The servo drive motor precisely controls the joint movement of the robotic arm to achieve precise adjustment of the position and posture of the gripping device.

[0100] Once the handling adjustment strategy is generated, a control signal needs to be sent to the servo drive motor according to the correction method and adjustment value in the strategy. For example, if the adjustment strategy is "move 30mm to the right", the control signal needs to drive the robotic arm to rotate to the right, causing the gripping device to move to the right. Specifically, after receiving the control signal, the servo drive motor will increase its output torque, causing the corresponding joint of the robotic arm (such as the shoulder or elbow joint) to rotate to the right, thereby causing the gripping device to move 30mm to the right in the horizontal direction, away from its current position. Assuming the current x-axis coordinate of the gripping device is 1000mm, after moving 30mm to the right, its x-axis coordinate will become 1030mm, thus achieving position correction.

[0101] If the adjustment strategy is "tilt back 6°", the control signal needs to cause the robotic arm to extend and retract backward, causing the hopper to tilt backward. At this time, the servo drive motor will reduce its output torque, controlling the end joint of the robotic arm (such as the wrist joint) to swing backward, causing the gripping device to tilt the hopper backward by 6°. For example, if the hopper's original pitch angle is 0°, after tilting back 6°, its pitch angle will become -6° (assuming the tilt direction is a negative angle), thus completing the attitude correction.

[0102] The adjustment value depends on the change in the output torque of the servo drive motor, and the two are directly proportional. For example, when the adjustment value is 10mm, the output torque of the servo drive motor needs to be increased by a certain amount to make the robotic arm rotate or extend by 10mm; if the adjustment value is 50mm, the torque increase will be correspondingly larger to drive the robotic arm to complete a 50mm displacement. This proportional relationship is achieved through a pre-set control parameter table or mathematical model, which is based on the kinematic characteristics of the robotic arm and the driving characteristics of the servo motor. For example, if an increase of 10 N·m in torque of a certain type of servo drive motor can make the robotic arm move 5mm in the x-axis direction, then when the adjustment value is 30mm, the motor torque needs to be increased by 60 N·m.

[0103] Taking a specific application scenario as an example: Suppose that during the handling process, the material bin's position continuously shifts to the left due to a localized ground protrusion. After analyzing the image data within the handling optimization area, the average deviation is determined to be 35mm, which is considered a moderate deviation. The resulting adjustment strategy is "shift 35mm to the right." At this point, the servo drive motor of the actuator receives the control signal and begins to increase its output torque. The motor drives the horizontal movement joint of the robotic arm to rotate to the right through gear transmission or lead screw transmission. As the torque gradually increases, the robotic arm drives the gripping device to slowly move to the right. When the displacement reaches 35mm, the servo drive motor stops outputting torque based on the feedback signal, completing the position correction.

[0104] For example, when the hopper is being gripped, a positional deviation of the gripping device causes it to tilt forward. The pitch angle in the handling optimization area deviates from the average value by 8°, resulting in an adjustment strategy of "tilting backward by 8°". After receiving the control signal, the servo drive motor reduces its output torque, driving the pitch joint of the robotic arm to swing backward. The gripping device at the end of the robotic arm tilts the hopper backward. When the angle sensor detects a pitch angle change of 8°, the motor stops, completing the attitude correction.

[0105] During the execution of the adjustment strategy, the servo drive motor needs to receive position or angle feedback signals in real time to ensure the accuracy of the adjustment value. Feedback signals are typically provided by encoders or sensors installed at the joints of the robotic arm. For example, encoders can measure the rotation angle of the robotic arm in real time, and force sensors can detect the force applied to the gripping device. When the feedback signal indicates that the preset adjustment value has been reached, the motor immediately stops to avoid over-adjustment. For example, during a 35mm rightward movement, the encoder feeds back the robotic arm's displacement data to the control system in real time. When the displacement reaches 35mm, the control system sends a stop signal to the servo drive motor, the motor stops increasing torque, and the robotic arm stops moving.

[0106] If the adjustment strategy includes both position correction and attitude correction, such as "move 20mm to the right and tilt 5° backward," the actuator needs to coordinate the movements of multiple joints of the robotic arm. The servo drive motors need to simultaneously control the movements of the horizontal movement joint and the pitch joint, so that the robotic arm moves 20mm to the right while tilting 5° backward. This process is achieved through multi-axis linkage control of the control system, ensuring that the two actions are completed synchronously and avoiding swaying or secondary deviation of the hopper due to the sequence of actions. For example, the control system decomposes the adjustment strategy into horizontal and pitch movement commands, which are sent to the corresponding servo drive motors. The two motors work together to make the robotic arm move and rotate along a predetermined trajectory.

[0107] The mechanical structure design of the actuator must meet the accuracy and stability requirements of the adjustment movements. For example, the joint clearance of the robotic arm needs to be controlled within a small range to avoid adjustment errors caused by clearance; the clamping force of the gripping device needs to be adjusted according to the weight and size of the material box to ensure that the material box will not fall off or deform during the gripping process. The selection of the servo drive motor must match the load and range of motion of the robotic arm. For example, when handling large material boxes, a high-torque motor should be selected to ensure sufficient driving force.

[0108] Furthermore, the actuators of different types of material handling equipment may vary. For example, the actuators on AGVs (Automated Guided Vehicles) may be integrated into the vehicle body, coordinating with the robotic arm's movements through the steering and movement of the wheels; while the actuators of fixed-station material handling equipment may be mounted on a gantry, achieving position adjustment through linear motion along the x, y, and z axes. Regardless of the structure, the core principle is that servo drive motors receive control signals to drive the robotic arm and gripping device to complete the correction methods and adjustment values ​​in the adjustment strategy.

[0109] Through the above steps, from receiving the adjustment strategy to sending control signals, and then to the servo drive motor driving the robotic arm and gripping device to execute actions, the precise adjustment of the material box's handling position and posture is ultimately achieved. In this process, the reliability of the mechanical structure, the control precision of the servo drive motor, and the real-time performance of the feedback system are all indispensable, jointly ensuring the accurate execution of the adjustment strategy and thus achieving precise material box handling control. In practical applications, the control parameters and mechanical structure need to be optimized according to the specific model of the handling equipment and the characteristics of the material box to adapt to different handling scenarios and adjustment requirements.

[0110] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0111] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A visual recognition bin handling control method, characterized by, The method comprises the following steps: During the material box handling operation, image data containing the position and attitude of the material box is acquired in real time by a visual acquisition device deployed in the handling area; The real-time acquired image data is compared with the pre-stored standard material box feature template, and the reference handling parameters of the material box in the current scene are determined in combination with the ambient light intensity, ground flatness and material box size parameters of the current handling scene; Differential analysis is performed on the real-time acquired image data and the reference handling parameters, abnormal areas whose position or attitude deviates beyond the tolerance range are identified, and the abnormal areas are marked as handling optimization areas that need to be adjusted; For the marked handling optimization areas, handling adjustment strategies are generated according to the change trend and deviation degree of the image data, the adjustment strategies including the specific ways and adjustment values of handling direction correction or attitude correction; The handling adjustment strategies are executed by the execution mechanism of the handling equipment to complete the precise handling control operation of the material box; Differential analysis is performed on the real-time acquired image data and the reference handling parameters, abnormal areas whose position or attitude deviates beyond the tolerance range are identified, and the abnormal areas are marked as handling optimization areas that need to be adjusted, and the specific steps are as follows: The real-time acquired continuous image sequence is compared with the reference handling parameters frame by frame, and the deviation value of the position and attitude of the material box in each image is calculated; The maximum tolerance range of the deviation value is set, and if the deviation value of a certain image exceeds the maximum tolerance range, the image is marked as an abnormal frame; The sliding window technology is used to process the marked abnormal frames, and the image interval in which the abnormal frames continuously appear is merged into a continuous abnormal area, which is the handling optimization area that needs to be adjusted; The maximum tolerance range of the deviation value is set, and if the deviation value of a certain image exceeds the maximum tolerance range, the image is marked as an abnormal frame, and the specific steps are as follows: By analyzing the fluctuation characteristics of the historical scene data in the standard material box feature template library, the upper limit and lower limit of the normal fluctuation of the deviation value are determined, and the interval between the upper limit and the lower limit is the maximum tolerance range of the deviation value; The deviation value of each frame of real-time acquired image is determined, and if the deviation value is greater than the upper limit of the normal fluctuation or less than the lower limit of the normal fluctuation, the frame is marked as an abnormal frame.

2. The visual identification of a bin handling control method according to claim 1, characterized in that, Real-time image data containing the position and attitude of the material box is acquired by a visual acquisition device deployed in the handling area, and the specific steps are as follows: Industrial cameras are arranged at the entrance of the handling channel, the working point of the handling equipment and the edge of the material box storage area, and each camera synchronously collects images at a fixed time interval; After the image information collected by each camera is converted into digital signals, it is transmitted to the main control processing unit through the communication module, and the main control processing unit restores the received digital signals to image data and stores them as a continuous image sequence in the order of acquisition time.

3. The visual identification of a bin handling control method according to claim 1, wherein, The real-time acquired image data is compared with the pre-stored standard material box feature template, and the reference handling parameters of the material box in the current scene are determined in combination with the ambient light intensity, ground flatness and material box size parameters of the current handling scene, and the specific steps are as follows: retrieve historical scene data with the same size of the current bin, the difference of ambient light intensity is not more than the set range and the difference of ground flatness is not more than the set range from the standard bin feature template library; statistically process the retrieved historical scene data, calculate the intermediate value and fluctuation interval of the bin handling position and attitude in the historical scene data, and take the intermediate value as the initial reference handling parameter; correct the initial reference handling parameter according to the current environment temperature and humidity, bin surface reflectivity and handling equipment initial position parameter to obtain the final reference handling parameter.

4. The visual identification of a bin handling control method according to claim 1, characterized in that, For the marked handling optimization area, generate a handling adjustment strategy according to the change trend and deviation degree of the image data, the adjustment strategy includes the specific way and adjustment value of handling direction correction or attitude correction, and the specific steps are as follows: analyze the change trend of the continuous image sequence in the handling optimization area, if the position coordinates present a continuous left shift trend, determine that the handling direction correction mode is right shift; if the attitude angle presents a continuous forward inclination trend, determine that the attitude correction mode is backward inclination; calculate the average value of the deviation value in the handling optimization area, and divide the deviation degree into three levels of slight, moderate and severe according to the size of the average value, each level corresponds to a preset adjustment value, and finally generate a handling adjustment strategy containing the correction mode and adjustment value.

5. The visual identification of a bin handling control method according to claim 1, wherein, Execute the handling adjustment strategy through the actuator arranged on the handling equipment to complete the precise handling control operation of the bin, and the specific steps are as follows: The handling equipment actuator includes a rotatable mechanical arm and a clamping device connected with the mechanical arm, and the actuator driving part is a servo driving motor matched with the mechanical arm; According to the correction mode and adjustment value in the handling adjustment strategy, control signals are sent to the servo driving motor, the servo driving motor drives the mechanical arm to rotate or stretch, and drives the clamping device to move, so as to change the handling position or attitude of the bin and realize the handling control of the bin.

6. The visual identification of a bin handling control method according to claim 4, characterized in that, Calculate the average value of the deviation value in the handling optimization area, and divide the deviation degree into three levels of slight, moderate and severe according to the size of the average value, and the specific steps are as follows: Calculate the sum of the deviation values of all abnormal frames in the handling optimization area, and divide it by the number of abnormal frames to obtain the average deviation value; If the average deviation value is less than the set slight deviation threshold, the deviation degree is slight; If the average deviation value is greater than or equal to the slight deviation threshold and less than the set severe deviation threshold, the deviation degree is moderate; If the average deviation value is greater than or equal to the severe deviation threshold, the deviation degree is severe.

7. The visual identification of a bin handling control method according to claim 5, wherein, According to the correction mode and adjustment value in the handling adjustment strategy, control signals are sent to the servo driving motor, and the specific steps are as follows: If the correction mode is right shift handling direction, the control signal is to increase the output torque of the servo driving motor to make the mechanical arm rotate to the right side and the clamping device move to the right side away from the current position; If the correction mode is backward inclination attitude correction, the control signal is to reduce the output torque of the servo driving motor to make the mechanical arm stretch to the rear and the clamping device make the bin incline backward; The adjustment value is realized by controlling the change amplitude of the output torque of the servo driving motor, and the torque change amplitude is in a positive proportional relationship with the adjustment value.

8. A visual recognition bin handling control system for implementing a visual recognition bin handling control method according to any one of claims 1 to 7, characterized by It comprises: A visual acquisition device is arranged in the conveying area to acquire image data containing the position and posture of the container in real time during the conveying operation of the container; A parameter determination module is configured to compare the image data acquired by the visual acquisition device in real time with a pre-stored standard container feature template, and determine the reference conveying parameters of the container in the current scene in combination with the ambient light intensity, ground flatness and container size parameters of the current conveying scene; An abnormality identification module is configured to perform difference analysis on the image data acquired by the visual acquisition device in real time and the reference conveying parameters determined by the parameter determination module, identify the abnormal area whose position or posture deviates beyond the tolerance range, and mark the abnormal area as a conveying optimization area that needs to be adjusted; A strategy generation module is configured to generate a conveying adjustment strategy according to the change trend and deviation degree of the image data for the conveying optimization area marked by the abnormality identification module, and the adjustment strategy includes the specific manner and adjustment value of the conveying direction correction or posture correction; An execution module is arranged in the conveying device to execute the conveying adjustment strategy generated by the strategy generation module and complete the precise conveying control operation of the container.

Images