A scene-based camera dynamic viewing method

Abstract

The application discloses a kind of scene-based camera dynamic observation method, belong to intelligent logistics robot visual perception technical field, method relies on mechanical arm J1 axis layout with coaxial rotating vertical rod and multilayer physical camera, constructs the logical camera mapping library of binding grasping posture, lifting height.This method first according to carriage size self-adapting configuration rotation angle sequence and stratified height threshold, follows xzy space priority scheduling;Transverse view angle can synchronous issue grasping and rotation instruction to realize parallel operation, vertical rely on height threshold stratified filtering target, switch logical camera layer by layer to complete global scanning.The application refines stratified granularity by logical camera, avoids carriage and mechanical arm shielding, eliminates view angle switching waiting time, relies on state machine control scanning process to avoid dead loop, multi-thread asynchronous operation balances detection and grasping rhythm, can adapt to different specifications carriage and goods, improves loading and unloading visual observation precision and overall operation efficiency.

Description

Technical Field

[0001] This invention relates to the field of visual perception technology for intelligent logistics robots, specifically a scene-based camera dynamic observation method. Background Technology

[0002] With the increasing application of artificial intelligence vision technology in industrial scenarios such as depalletizing, loading and unloading, vision perception systems have become a core module for improving target detection accuracy, operational stability and execution efficiency. The camera's installation structure and field-of-view switching control method are key factors that determine the operational performance of the vision system.

[0003] Existing visual observation solutions for loading and unloading scenarios already possess the functions of vertical layered observation and lateral viewpoint switching. Vertical layering is mostly achieved using independent physical cameras as the basic unit, while lateral field of view coverage is achieved through gimbal rotation or fixed deployment of multiple cameras. Camera installation generally adopts an "eye on hand" or "eye outside hand" deployment method. Since vertical layering is directly bounded by the field of view of the entire physical camera, the layering granularity is relatively coarse, which cannot match the fine spatial grasping sequence. This easily leads to the misidentification of targets in the current observation layer. At the same time, conventional camera installation methods are limited by the deployment location and are easily obstructed by the robotic arm structure or the side wall of the vehicle, resulting in insufficient observation stability. In terms of the control logic of viewpoint switching, existing solutions all adopt a serial execution scheduling method. The grasping operation can only be started after the viewpoint adjustment and photo detection are all completed. The viewpoint switching process cannot be carried out in parallel with the grasping action, which adds extra waiting time and ultimately leads to low overall operation efficiency. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing camera observation solutions in loading and unloading scenarios by providing a scene-based dynamic camera observation method. This method optimizes the camera installation architecture and establishes a logical camera mapping mechanism, refining the granularity of vertical observation layers. It avoids occlusion of the camera's field of view by the robotic arm structure and the vehicle space, reducing misidentification caused by cross-layer target intrusion and improving the stability and accuracy of visual observation. Simultaneously, by using parallel scheduling logic for grasping operations and perspective switching, it eliminates the independent waiting time during perspective switching, shortening the single-cycle operation. Furthermore, relying on scene-adaptive parameter configuration capabilities, it enhances the method's adaptability to different sized operating scenarios. Ultimately, while ensuring full-range field of view coverage, it effectively improves the overall operational efficiency of the loading and unloading vision system.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a scene-based camera dynamic observation method, specifically including the following steps: S1 Camera Installation Architecture: A vertical support rod with left and right rotation axes is fixed on the J1 axis of the robotic arm. Multiple physical cameras are arranged vertically along the rod. A logical camera mapping library is established, binding the physical cameras with grasping postures and lifting heights in multiple dimensions. Each physical camera is divided into multiple independent observation units. This installation architecture relies on the fixed rotating base of the robotic arm's J1 axis as the main support, ensuring that the camera as a whole maintains a stable spatial reference along the J1 axis. This avoids field-of-view obstruction caused by the movement of the robotic arm's end effector or external supports. At the same time, the vertical support rod and the left and right rotation axes are integrated, making lateral rotation and vertical layered observation independent and non-interfering. The logical camera mapping library divides the observation space of the same physical camera under different working postures. Without increasing hardware costs, the originally single physical observation unit is decomposed into multiple logical observation units corresponding to different working heights and grasping postures. This allows the granularity of vertical observation layering to match the layering accuracy of goods stacking, providing the hardware and mapping foundation for strictly following the spatial grasping sequence.

[0006] S2 adaptive configuration for work scenarios: Based on the width and vertical stacking height of the carriage to be worked on, corresponding left and right rotation angle sequences and layer height thresholds are matched for each logical camera, generating observation and grasping linkage rules sorted by xzy spatial priority. This configuration process automatically generates angle combinations and height boundaries that can completely cover the work space by matching and calculating scene geometric parameters with the camera's inherent field of view parameters, ensuring that the observation range has neither blind spots nor excessive overlap. At the same time, the xzy spatial priority linkage rules deeply bind the spatial order of observation scheduling and grasping execution, ensuring that visual observation always prioritizes covering the target area of ​​the current grasping operation, avoiding time loss caused by invalid field of view scanning.

[0007] S3 Dynamic Left and Right View Scheduling: This system executes angle rotation and image detection sequentially from the center outwards, determining the target distribution within the current viewpoint in real time. When the active rotation trigger condition is met, a rotation command is simultaneously sent to the robotic arm to perform the grasping task, enabling parallel execution of the grasping operation and viewpoint switching. This scheduling logic, by judging the spatial distribution of the grasped target in real time, predicts the timing of the next viewpoint switch in advance, moving the start point of the rotation action forward to the grasping action execution phase. It utilizes the robotic arm's grasping time window to complete viewpoint adjustment and positioning, eliminating the need for a separate waiting period for viewpoint switching. The scanning sequence from the center outwards aligns with the inside-out approach to cargo grasping, enabling rapid target location in the core work area during the first observation, shortening the initial grasping response time.

[0008] S4 Vertical View Layered Iteration: When there are no valid targets to be captured above the height threshold corresponding to the current logical camera and at least one round of full-angle scanning has been completed, the camera switches vertically downwards to the next logical camera, resets the height threshold, and repeats step S3 until all vertical observation layers have been traversed. This layered iteration mechanism uses the height threshold as the spatial boundary of each observation layer. Combined with the completion judgment of full-angle scanning, it ensures that all graspable targets in the current layer are identified and processed before entering the next layer, effectively avoiding the confusion of the grasping order caused by lower-layer targets entering the observation range in advance. The mechanism of resetting and updating the height threshold with the switching of layers can adapt to the actual stacking state of goods in each layer, ensuring that the layer boundary matches the actual top surface height of the goods.

[0009] Furthermore, the active rotation triggering condition in step S3 specifically satisfies any one of the following: the user configures a forced angle-by-angle rotation mode, the current angle is the middle and there is no target to grab in the central area, there are no remaining targets to grab or the remaining targets can be grabbed in one go and the user enables the early rotation permission; each triggering condition is judged sequentially according to a preset priority to avoid repeated triggering and rotation action conflicts. The multi-condition hierarchical judgment mechanism can cover scenarios with different job configurations and target distributions. It supports users to forcibly set the angle-by-angle rotation job mode according to job requirements, and can also automatically execute the optimal switching time according to the target distribution. The priority sorting design can avoid instruction conflicts when multiple conditions are met at the same time, ensuring the stable operation of the rotation scheduling logic. At the same time, the active early rotation judgment rule can maximize the use of the parallel window of the grabbing job, further reducing the overall job time.

[0010] Furthermore, the rules for resetting and setting the height threshold in step S4 are as follows: when the camera rotates back to its initial angle, the height threshold of the current logical camera is reset; when the height threshold is in a reset state, the height threshold for this round is regenerated based on the center height of the target captured in the current batch minus a preset offset; when the difference in the size of the work box exceeds the preset threshold, the offset of the height threshold is automatically adaptively fine-tuned. The mechanism of resetting the height threshold with the return of the initial angle ensures that each horizontal scan is based on the actual height of the goods at the current level, avoiding the filtering deviation caused by the height parameters left over from the previous scan in the new round of observation; the setting method based on the target center height minus the offset can accommodate the height fluctuations of goods in the same layer, avoiding the target omission caused by the unevenness of the top surface of the goods, while the adaptive fine-tuning under the difference in box size can automatically adjust the looseness of the layer boundary according to the fluctuation of the goods size, improving the adaptability of the method to goods of different specifications.

[0011] Furthermore, in the logical camera mapping library, the same physical camera is treated as an independent logical camera unit when it corresponds to different grasping postures and base lifting heights. Each logical camera unit pre-stores the field of view boundary parameters and lens distortion correction coefficients under the corresponding viewpoint for accurate conversion of target spatial coordinates. Dividing the same physical camera under different grasping postures and lifting heights into independent logical units ensures that each set of observation postures corresponds to a clear spatial observation range, avoiding blurring of the field of view boundary due to posture changes. The pre-stored field of view boundary parameters and lens distortion correction coefficients can be directly used for the conversion calculation from target pixel coordinates to spatial three-dimensional coordinates without recalibrating after each photo, which improves the accuracy of coordinate conversion and reduces the calculation time for a single round of observation.

[0012] Furthermore, in step S1, the vertical support and the left and right rotation axes are both fixed on the rotation base of the robotic arm's J1 axis. During the photo-taking process, the other joints of the robotic arm, except for the J1 axis, can move freely. The rotation center of the J1 axis and the rotation center of the left and right rotation axes are coaxially arranged to eliminate the target position offset caused by the viewpoint switching. The coaxial arrangement of the J1 axis and the left and right rotation axes ensures that the rotation center of the lateral viewpoint rotation coincides with the rotation center of the robotic arm's base. During the viewpoint rotation, the spatial reference position of the camera remains constant, and no additional translational offset occurs. This eliminates the target coordinate system deviation caused by the viewpoint switching, reduces the computational complexity of subsequent coordinate transformation, and allows the other joints to move freely during the photo-taking process. This enables the robotic arm to complete pose adjustment or auxiliary actions while visually observing, further improving the parallelism of the operation.

[0013] Furthermore, in step S2, the left and right rotation angle sequence supports custom configuration, with the camera facing forward as the 0-degree reference, left rotation being positive and right rotation being negative; the system can automatically calculate and generate the optimal number of angle sets and corresponding angle values ​​covering the entire width of the carriage based on the input carriage width parameter. The positive and negative angle definition method with the front facing forward as the 0-degree reference conforms to conventional rotation control logic, facilitating user debugging and parameter configuration; the function of automatically calculating the optimal number of angle sets and values ​​based on the carriage width eliminates the process of manual angle-by-angle debugging, and users only need to input scene size parameters to obtain a complete angle configuration scheme, while ensuring that carriages of different widths can obtain complete field of view coverage, avoiding blind spots or excessive overlap on both sides.

[0014] Furthermore, in step S4, when switching the logic camera downwards, the left and right rotation axes must first be returned to their initial angles. When switching to the logic camera with the corresponding forward gripping posture, the trolley carrying the robotic arm is simultaneously controlled to move forward a preset distance. The moving distance can be adaptively adjusted according to the box size and the depth parameters of the carriage. The setting of returning to the initial angle before vertical switching ensures that the lateral scanning of each level starts from a unified reference pose, so that the observation process of each level maintains a consistent scheduling logic, which is convenient for program control and status management. The design of synchronously controlling the trolley to move forward when switching to the forward gripping posture can cooperate with the forward movement of the camera's observation range, so that the working depth gradually advances with the gripping process, adapting to the loading and unloading rhythm of the carriage from the outside to the inside. The adaptive adjustment of the moving distance can match the optimal step size according to the cargo size and the depth of the carriage, ensuring that the observation range after each forward movement just covers the area to be worked.

[0015] Furthermore, when a full-angle scan is completed under the current logical camera and no target is captured after returning to the initial angle, a vertical view switching is triggered. The scanning completion status of each logical camera is marked by a state machine to avoid repeated loop scanning within the same level. The rule of triggering vertical switching only after a full-angle scan is completed ensures that all lateral angle areas within the current level have completed target screening, avoiding target omissions caused by premature level switching due to no target on one side. The state machine marking method allows for unified management of the scanning completion status of each logical camera, clearly recording the work progress of each level, effectively preventing the problem of infinite loop scanning within the same level, and improving the stability and reliability of the scheduling logic.

[0016] Furthermore, when switching to the last logical camera and completing a full-angle scan without capturing a target, the current scene observation operation is considered complete. Simultaneously, the status flags, height thresholds, and rotation angles of all logical cameras are reset, returning the system to its initial standby pose. This unified reset mechanism after full-level traversal restores all state parameters and hardware poses to their initial standby state. This facilitates system reset after a single operation and provides a unified initial reference for the next operation, preventing parameter residues from the previous operation from interfering with subsequent operations and improving the reliability of continuous system operation.

[0017] Furthermore, image detection and perspective switching are executed by independent threads, and the generated grasping targets are stored in a task queue for the robotic arm to schedule. When the grasping queue is empty, a new round of image detection is automatically triggered, realizing the decoupling and asynchronous operation of the observation and grasping processes. The design of running image detection and perspective switching by independent threads can decouple the visual observation process from the robotic arm grasping process. The two do not need to be strictly synchronized. The grasping task queue acts as an intermediate buffer, which can balance the speed difference between visual detection and robotic arm execution. The mechanism of automatically triggering a new round of detection when the grasping queue is empty can ensure that the robotic arm always has tasks to execute, avoiding the window period where the robotic arm waits for images, and further improving the overall smoothness and execution efficiency of the system.

[0018] Compared with existing technologies, the present invention provides a scene-based camera dynamic observation method, which has the following beneficial effects: To address the shortcomings of existing vertical layering methods and the ease with which installation structures can obstruct view, this invention employs a robotic arm with a vertical pole featuring left and right rotation axes mounted on its J1 axis, and a vertically arranged structure of multiple physical cameras. Combined with a logical camera mapping mechanism that binds the physical cameras to the grasping posture and lifting height, this approach not only prevents the cameras from being obstructed by the robotic arm structure and the vehicle space, thus improving the stability of the observation process, but also enables more refined vertical observation layering. This ensures that the grasping rules are executed according to the xzy spatial priority, effectively reducing the problem of misidentification caused by cross-layer target mixing.

[0019] To address the shortcomings of existing methods that involve sequential execution of viewpoint switching and long operation times, this invention employs a parallel scheduling logic that synchronously issues grasping task instructions and viewpoint rotation instructions. This allows the viewpoint switching action and the robotic arm's grasping operation to overlap in time, eliminating the independent waiting time for viewpoint switching, effectively shortening the single-cycle operation, and improving the overall operation efficiency of the system.

[0020] This invention can adaptively configure the left and right rotation angle sequence and layer height threshold according to the operation scenario. Combined with the observation process of left and right dynamic scheduling and vertical layer iteration, it can adapt to operation scenarios with different widths and stacking heights. While ensuring full field of view coverage, it further improves the scenario adaptability and scheduling flexibility of the method. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the overall process flow of a scene-based camera dynamic observation method according to the present invention. Figure 2 This is a system architecture block diagram of a scene-based camera dynamic observation method according to the present invention; Figure 3 This is a flowchart illustrating the left and right perspective switching logic branches of a scene-based camera dynamic observation method according to the present invention. Detailed Implementation

[0022] Please see Figure 1-3 This invention provides a scene-based camera dynamic observation method, which is applied to an intelligent loading and unloading robotic arm system with 3D visual perception capabilities to achieve orderly visual observation and grasping scheduling of goods within the confined space of a cargo compartment. The following describes the specific implementation process of this method in detail with reference to the steps and flowchart.

[0023] S1 Camera Mounting Architecture: A vertical support rod with left and right rotation axes is fixed on the J1 axis of the robotic arm. Multiple physical cameras are arranged vertically along the rod. A logical camera mapping library is established, binding the physical cameras to grasping posture and lifting height in multiple dimensions. Each physical camera is divided into multiple independent observation units. In specific implementation, the J1 axis of the robotic arm is the bottom rotary joint, capable of rotating around the vertical axis. The bottom of the vertical support rod is fixedly installed on the rotary base of the J1 axis, so that the vertical support rod rotates synchronously with the rotation of the J1 axis. The left and right rotation axes are set on the rod body of the vertical support rod. The rotation axis of the rotation axis is set vertically and is coaxial with the rotation center of the J1 axis, that is, the vertical rotation axes of the two are completely coincident, thereby eliminating the target position offset caused by the switching of the viewpoint. Multiple physical cameras are sequentially fixed along the vertical column, with each camera's lens facing the working side. They can rotate around the vertical axis along with the left and right rotation axes, allowing for adjustment of the lateral viewing angle. This mounting structure relies on the fixed rotating base of the robotic arm's J1 axis as the main support, ensuring the camera maintains a stable spatial reference along the J1 axis. This avoids obstruction of the field of view caused by the robotic arm's end effector movement or external supports. Furthermore, the integrated layout of the vertical column and the left and right rotation axes ensures that lateral rotation and vertical layered observation are independent and do not interfere with each other. During the photography operation, the robotic arm's joints, except for the J1 axis, can move freely, requiring only the J1 axis to remain locked. This ensures a stable camera observation reference while allowing the robotic arm to simultaneously perform auxiliary actions such as pose adjustment and cargo handling, improving the parallelism of the operation. The specific construction process of the logical camera mapping library is as follows: For each physical camera, the field of view is calibrated in advance under different grasping posture types and different base lifting heights. The grasping posture types include at least two types: side grasping posture and front grasping posture, which correspond to different grasping methods in loading and unloading operations. The base lifting height corresponds to the lifting adjustment gear of the robotic arm base or pole itself, and each gear corresponds to a specific height value. For each combination of "physical camera number + grasping posture type + lifting height value", calibration is performed to obtain the camera's effective observation space range, vertical field of view boundaries, horizontal field of view boundaries, and lens distortion correction parameters in that state. Each combination is defined as an independent logical camera unit and assigned a unique logical camera identifier. The identifiers and corresponding parameters of all logical camera units together form a logical camera mapping library, stored in the system's non-volatile storage unit for subsequent observation and scheduling processes. By dividing the observation space of the same physical camera under different working postures, the logical camera mapping library decomposes the originally single physical observation unit into multiple logical observation units corresponding to different working heights and grasping postures without increasing hardware costs. This allows the granularity of vertical observation layering to match the layering accuracy of cargo stacking, providing the hardware and mapping foundation for strictly adhering to the spatial grasping sequence. The field-of-view boundary parameters and lens distortion correction coefficients pre-stored in each logical camera unit can be directly used for subsequent conversion calculations from target pixel coordinates to three-dimensional spatial coordinates. The specific conversion process is as follows: the original image or point cloud data acquired by the camera is first distorted by the pre-stored distortion correction coefficients to eliminate the coordinate deviation caused by lens distortion. Then, combined with the internal and external parameters corresponding to the logical camera, the target position in the pixel coordinate system is converted into three-dimensional spatial coordinates in the world coordinate system. There is no need to recalibrate the camera after each photo, which improves the accuracy of coordinate conversion and reduces the calculation time of a single round of observation.

[0024] S2 adaptive configuration for operational scenarios: Based on the width and vertical stacking height of the cargo compartment to be operated, the system matches corresponding left and right rotation angle sequences and layer height thresholds for each logical camera, generating observation and grasping linkage rules sorted by xzy spatial priority. In specific implementation, the system can obtain the width and vertical stacking height parameters of the cargo compartment to be operated through user input or front-mounted sensor detection. The width parameter corresponds to the lateral dimension inside the compartment, and the vertical stacking height corresponds to the total height of the stacked goods. For the configuration of the left and right rotation angle sequence, the system pre-stores the horizontal field of view parameters of each physical camera at the standard working distance. Combining the current camera installation distance and the input total width of the cargo compartment, the system calculates the lateral width that the camera can cover at a single angle. Then, with the constraint of complete coverage of the cargo compartment width without blind spots, the system calculates the minimum number of angle sets required and generates the corresponding angle value sequence according to the principle of left-right symmetry. The angle values ​​are defined with the camera facing forward as the 0-degree reference. Rotation to the left is a positive angle, and rotation to the right is a negative angle. The generated angle sequence can be manually fine-tuned by the user according to the actual work situation. The final confirmed angle sequence is bound and stored with the current logical camera. This configuration process automatically generates angle combinations and height boundaries that completely cover the work space by matching scene geometric parameters with the camera's inherent field of view parameters. This ensures that the observation range has neither blind spots nor excessive overlap, reducing invalid scanning areas. The positive and negative angle definition method, with the front as the 0-degree reference, conforms to conventional rotation control logic, facilitating user debugging and parameter configuration. The system automatically calculates the optimal number and values ​​of angle groups based on the width of the cargo compartment, eliminating the need for manual angle-by-angle adjustments. Users only need to input scene size parameters to obtain a complete angle configuration scheme, ensuring full field of view coverage for cargo compartments of different widths and avoiding blind spots or excessive overlap on both sides. Regarding the configuration of layered height thresholds, the system assigns a corresponding height range to each logical camera in descending order based on the total stacked height of the goods and the vertical observation range of each logical camera in the logical camera mapping library. Each logical camera corresponds to a height threshold, which represents the lowest height boundary that the current logical camera is responsible for observing. All height thresholds decrease sequentially from top to bottom, forming a layered vertical observation system. Based on this, the system generates observation and capture linkage rules sorted by xzy spatial priority, where the x-direction corresponds to the horizontal left-right direction of the cargo compartment, the z-direction corresponds to the vertical height direction, and the y-direction corresponds to the front-back depth direction of the cargo compartment. The xzy priority is as follows: prioritize advancing the operation from front to back along the y direction; prioritize processing the upper layers in the z direction at the same front and back positions; prioritize processing targets in the middle area of ​​the x direction at the same height level, and then gradually expand to both sides. The order of observation scheduling is consistent with the order of grasping execution. Visual observation always prioritizes covering the target area of ​​the current grasping operation, ensuring that the identified target matches the priority of the current grasping operation and avoiding time loss caused by invalid field scans. The linkage rule of xzy spatial priority deeply binds the spatial order of observation scheduling and grasping execution, so that visual perception and operation execution form an orderly connection, ensuring that loading and unloading operations proceed stably according to the established spatial order.

[0025] S3 Left and Right View Dynamic Scheduling: The system executes angle rotation and image detection sequentially from the center outwards, determining the target distribution within the current viewpoint in real time. When the active rotation trigger condition is met, a rotation command is simultaneously issued to the robotic arm to perform the grasping task, enabling parallel execution of the grasping operation and viewpoint switching. In specific implementation, after the system initiates lateral scanning, it first controls the left and right rotation axes to rotate to the initial angle in the angle sequence. The initial angle is typically 0 degrees directly in front, corresponding to the middle area of ​​the carriage. Upon reaching this position, it triggers the existing image detection, and the camera acquires 3D point cloud data from the current viewpoint. Through point cloud preprocessing and target segmentation and recognition algorithms, all cargo targets within the current field of view are extracted and converted into 3D coordinates in the world coordinate system. After target extraction, targets meeting the current height threshold are stored in the grasping task queue. After completing the image capture and target extraction at the current angle, the system rotates sequentially from the center outwards in both positive and negative angle directions, performing image detection at each angle until all angles in the angle sequence have been traversed.

[0026] The scanning sequence from the center outwards aligns with the inside-out approach to cargo handling, enabling rapid location of the target in the core work area during the initial observation and shortening the initial gripping response time. The active rotation trigger conditions include three items. Meeting any one of these conditions will trigger the pre-rotation action, and the three conditions are determined sequentially according to preset priorities to avoid repeated triggering and conflicts with rotation actions. The priorities, from highest to lowest, are as follows: The first item is a user-configured forced angle-by-angle rotation mode. When the user enables forced rotation mode in the system configuration, after completing a gripping task at one angle, the rotation action at the next angle is forcibly triggered. This mode is suitable for scenarios where targets are evenly distributed and require rotating operations at fixed angles. The second condition is that the current angle is in the middle and there is no target to grab in the central area. When the system determines that it is currently in the middle angle and there is no target to grab in the central area of ​​the current field of view, it triggers a rotation to the two sides. This condition is used to adapt to scenarios where the target distribution is uneven and the central area is grabbed first. The third condition is that there are no remaining targets to be grasped, or the remaining targets can be grasped in one go and the user has enabled the advance rotation permission. When there are no remaining targets to be grasped at the current angle, or the number and position of the remaining targets can be completed in one grasping action, and the user has enabled the advance rotation permission in advance, the view rotation action is triggered. The multi-condition hierarchical judgment mechanism can cover different job configurations and target distribution scenarios. It supports users to force the setting of a job mode that rotates angle by angle according to job requirements, and can also automatically execute the optimal switching time according to the target distribution. The priority sorting design can avoid instruction conflicts when multiple conditions are met at the same time, ensuring the stable operation of the rotation scheduling logic. At the same time, the judgment rule of active advance rotation can maximize the use of the parallel window of the grasping job, further reducing the overall job time. The specific implementation of the parallel execution of grasping and rotation is as follows: The system controller adopts a multi-task parallel scheduling mechanism. When the active rotation condition is met, the controller sends the grasping motion command of the current batch of targets to the robotic arm motion control module and the angle rotation command to the rotation axis drive module in the same control cycle. The two commands carry the same trigger timestamp to ensure that the two actions start at the same time. Because the robotic arm's grasping motion path is longer and takes longer, while the rotation axis's angle adjustment takes less time, the rotation can be completed within the robotic arm's grasping time window. By the time the robotic arm completes the current grasping motion and returns to the waiting position, the camera has already completed its angle adjustment and is ready to start the next image detection, eliminating the need for a separate waiting period for perspective switching. This achieves parallel overlap between grasping operations and perspective switching in time, effectively shortening the total time of a single round of operation. If there is no target to grasp at the current angle, the system performs angle rotation for targets. This rotation is limited to the same vertical level corresponding to the current logical camera, meaning only the left and right angles are switched, without vertical camera switching. When all angles in the angle sequence have been traversed and the rotation axis returns to the initial angle, in order to avoid getting stuck in an infinite loop of scanning within the same level, the system will trigger a vertical level switching decision process and will no longer continue to repeat the horizontal scanning.

[0027] S4 Vertical View Layer Iteration: When there are no valid targets above the height threshold corresponding to the current logical camera and at least one round of full-angle scanning has been completed, the system switches to the next logical camera vertically downwards, resets the height threshold, and repeats step S3 until all vertical observation layers have been traversed. In specific implementation, the height threshold serves as the spatial boundary for vertical layering, used to filter out targets below the current layer to prevent lower-layer targets from mixing into the current layer and causing confusion in the capture order. The management of the height threshold includes three stages: initialization, filtering, and resetting. First, the height threshold is initialized. When the camera rotates back to the initial angle, it means that a new round of horizontal scanning is about to begin. The height threshold used in the previous round is no longer suitable for the current round. At this time, the system resets the height threshold corresponding to the current logical camera to an invalid state, that is, height filtering is not enabled. When the system program first starts, all height thresholds are reset to an invalid state to ensure that the first round of scanning is not limited by preset parameters. When the height threshold is in an effective setting state, the system compares the center height coordinates of each identified cargo target with the current height threshold, retaining only targets with a center height greater than or equal to the height threshold as valid grasping targets. Targets below the height threshold are filtered out and do not enter the grasping task queue for the current round, thus ensuring that the current level only processes cargo above the corresponding height. When the height threshold is in an invalid reset state, the system does not perform height filtering, includes all identified targets in the current view in the candidate range, and simultaneously initiates the height threshold reset process. The height threshold reset rule is as follows: the system counts the center height values ​​of all valid grasping targets identified in the current batch, takes the baseline height value, subtracts the preset height offset, and uses the calculation result as the height threshold for this round. The reference height value can be the maximum center height of the current batch of targets or the average center height of all targets. The height offset is preset based on the height dimensions of standard goods to accommodate height fluctuations of goods in the same layer, avoiding missed detections due to uneven placement of goods. When the difference in box dimensions of the current batch of goods exceeds a preset proportional threshold, the system automatically and adaptively adjusts the height offset by increasing the value to expand the height tolerance range of the same layer, preventing false filtering of targets in the same layer due to inconsistent box heights. The mechanism of resetting the height threshold with the initial angle regression ensures that each horizontal scan is based on the actual height of the goods in the current layer, avoiding filtering deviations caused by height parameters left over from the previous scan. The method of setting the target center height by subtracting the offset can accommodate the height fluctuations of goods in the same layer, avoiding the omission of targets due to uneven top surfaces of goods. The adaptive fine-tuning under the difference in box size can automatically adjust the looseness of the layer boundary according to the fluctuation of the cargo size, improving the adaptability of the method to goods of different specifications.

[0028] When the system determines that there are no valid targets above the height threshold corresponding to the current logical camera, and at least one complete full-angle lateral scan has been completed, it triggers the operation of switching the logical camera downwards. When switching logical cameras, the system first controls the left and right rotation axes to return to their initial angles, ensuring that the next level of scanning starts from a unified reference pose. Then, it calls the logical camera mapping library and switches to the next logical camera unit in a top-to-bottom order, loading all parameters corresponding to that logical camera, including field-of-view parameters, distortion parameters, and default height range. If the switched logical camera corresponds to a positive grasping posture, the system synchronously controls the trolley carrying the robotic arm to move a preset distance into the cargo compartment, allowing the camera's observation range to advance into the cargo compartment as the work level decreases, adapting to the loading and unloading rhythm from the outside in. The travel distance of the trolley can be manually configured by the user based on the depth of a single row of goods, or it can be calculated by evenly distributing the distance based on the input total depth and total number of layers of the carriage. After the trolley reaches its destination, the drive module sends a signal to the system to indicate that it has reached its destination, and the system then starts the lateral scan of the next level. When switching to a logic camera downwards, the system first returns to the initial angle setting, ensuring that the lateral scan of each level starts from a unified reference pose. This maintains a consistent scheduling logic for the observation process of each level, facilitating program control and state management. The design of synchronously controlling the trolley's forward movement when switching to the forward gripping posture can coordinate with the forward movement of the camera's observation range, allowing the working depth to gradually advance with the gripping process. This adapts to the loading and unloading rhythm of the cargo compartment from the outside in. The adaptive adjustment of the travel distance can match the optimal step size according to the cargo size and the depth of the cargo compartment, ensuring that the observation range after each forward movement just covers the area to be worked.

[0029] The system uses a finite state machine to mark and manage the scanning completion status of each logic camera. The state machine includes six states: initial standby state, horizontal scanning state, angle rotation state, height determination state, vertical switching state, and task completion state. After the system starts up, it enters the initial standby state. Upon receiving the job start command, it jumps to the horizontal scanning state to perform image detection and target extraction at the current angle. When the rotation trigger condition is met, the system jumps to the angle rotation state and executes the rotation axis drive control. After rotating to the correct position, it returns to the horizontal scanning state. When a full-angle scan is completed and it is determined that there are no remaining targets at the current height, the system jumps to the height determination state to confirm whether it is necessary to switch the vertical level. If a switch is required, it enters the vertical switching state, performs logical camera switching and parameter reset, and returns to the horizontal scanning state after the switch is completed. If all logical cameras have been traversed and no targets remain, the system enters the job completion state, performs a system reset, and returns to the initial standby state. Through state machine state transition constraints, the job progress at each level can be clearly recorded, effectively preventing infinite loops within the same level and improving the stability and reliability of the scheduling logic.

[0030] When the system switches to the last logical camera and completes a full-angle scan under that camera without successfully capturing a target, the observation task for the current scene is considered complete. After the task ends, the system performs a unified reset operation, which includes: restoring the state flags of all logical cameras to their initial unexecuted state, resetting all height thresholds to an invalid state, returning the left and right rotation axes to their initial 0-degree angle, returning the trolley to its initial working position, clearing any remaining tasks in the capture task queue, and finally returning the entire system to its initial standby pose, awaiting the next task start command. This unified reset mechanism after full-level traversal restores all state parameters and hardware poses to their initial standby state. This facilitates system reset after a single task and provides a unified initial reference for the next task, preventing parameter residue from the previous task from interfering with subsequent tasks and improving the reliability of continuous system operation.

[0031] Throughout the observation and operation process, photo detection and perspective switching are performed by a separate photo detection thread, which is decoupled from the robotic arm execution thread and runs asynchronously. The image detection thread is responsible for performing operations such as angle rotation control, camera image triggering, point cloud data processing, target recognition and segmentation, height filtering, and target sorting and queuing according to the above scheduling logic. The identified valid grasping targets are sorted according to the xzy priority rule and stored in the first-in-first-out task queue. The robotic arm execution thread continuously monitors the status of the task queue. When the queue is not empty, it retrieves the target task at the head of the queue, plans the motion path, and executes the grasping operation. When the grasping queue is empty, the robotic arm execution thread sends a new round of detection requests to the image detection thread. Upon receiving the request, the image detection thread immediately starts the next image detection, quickly replenishing the targets in the task queue. The design of having image detection and perspective switching run in separate threads decouples the visual observation process from the robotic arm grasping process. The two do not need to be strictly synchronized. The grasping task queue acts as an intermediate buffer, balancing the speed difference between visual detection and robotic arm execution. The mechanism of automatically triggering a new round of detection when the capture queue is empty ensures that the robotic arm always has tasks to perform, avoids the window period when the robotic arm is waiting to take pictures, and further improves the overall smoothness and execution efficiency of the system.

Claims

1. A scene-based camera dynamic observation method, characterized in that, Includes the following steps: S1 camera installation architecture setup: A vertical pole with left and right rotation axes is fixed on the J1 axis of the robotic arm. Multiple physical cameras are arranged vertically along the pole. A logical camera mapping library is established that binds the physical cameras to the grasping posture and lifting height in multiple dimensions. Each physical camera is divided into multiple independent observation units. S2 adaptive configuration for work scenarios: Based on the width and vertical stacking height of the carriage to be worked on, match the corresponding left and right rotation angle sequence and layer height threshold for each logical camera, and generate observation and grasping linkage rules sorted by xzy space priority; S3 left and right view dynamic scheduling: Execute angle rotation and photo detection in the order from the middle to both sides, determine the target distribution in the current view in real time, and when the active rotation trigger condition is met, issue a rotation command at the same time as issuing the grasping task to the robotic arm, so as to realize the parallel execution of grasping operation and view switching. S4 Vertical View Layer Iteration: When there is no effective target captured above the height threshold corresponding to the current logical camera and at least one round of full-angle scanning is completed, switch to the next logical camera vertically downward, reset the height threshold and repeat step S3 until all vertical observation layers have been traversed. The active rotation triggering condition mentioned in step S3 is specifically to meet any one of the following: the user configures a forced angle-by-angle rotation mode, the current angle is the middle and there is no target to grab in the center area, there are no remaining targets to grab or the remaining targets can be grabbed in one go and the user enables the early rotation permission; each triggering condition is judged in sequence according to the preset priority to avoid repeated triggering and conflict with the rotation action; The rules for resetting and setting the height threshold in step S4 are as follows: when the camera rotates back to the initial angle, the height threshold of the current logical camera is reset; when the height threshold is in the reset state, the height threshold of this round is regenerated based on the center height of the target captured in the current batch minus the preset offset; when the difference in the size of the work box exceeds the preset threshold, the offset of the height threshold is automatically adaptively fine-tuned. In the logical camera mapping library, the same physical camera is treated as an independent logical camera unit when it corresponds to different grasping postures and base lifting heights. Each logical camera unit pre-stores the field of view boundary parameters and lens distortion correction coefficients under the corresponding viewpoint for accurate conversion of target space coordinates.

2. The scene-based camera dynamic observation method according to claim 1, characterized in that, In step S1, the vertical support and the left and right rotation axes are both fixed on the rotation base of the robotic arm J1 axis. During the photo taking process, the other joints of the robotic arm, except for the J1 axis, can move freely. The rotation center of the J1 axis and the rotation center of the left and right rotation axes are set coaxially to eliminate the target position offset caused by the switching of the viewing angle.

3. The scene-based camera dynamic observation method according to claim 1, characterized in that, In step S2, the left and right rotation angle sequence supports custom configuration, with the camera facing forward as the 0-degree reference, left rotation being positive and right rotation being negative; the optimal number of angle groups and corresponding angle values ​​covering the entire width of the carriage are automatically calculated and generated based on the input carriage width parameters.

4. The scene-based camera dynamic observation method according to claim 1, characterized in that, In step S4, when switching the logic camera downwards, the left and right rotation axes must first be returned to their initial angles; when switching to the logic camera with the corresponding positive gripping posture, the trolley carrying the robotic arm is simultaneously controlled to move forward a preset distance; the moving distance can be adaptively adjusted according to the box size and the depth parameters of the carriage.

5. The scene-based camera dynamic observation method according to claim 1, characterized in that, When the current logical camera completes a full-angle scan and returns to the initial angle without capturing a target, a vertical viewpoint switch is triggered; the scanning completion status of each logical camera is marked by a state machine to avoid repeated loop scanning within the same level.

6. The scene-based camera dynamic observation method according to claim 1, characterized in that, When the last logical camera is switched to and a full-angle scan is completed without capturing a target, the current scene observation operation is determined to be over; the status flags, height thresholds and rotation angles of all logical cameras are reset simultaneously, so that the system returns to the initial standby pose.

7. The scene-based camera dynamic observation method according to claim 1, characterized in that, The photo detection and perspective switching are executed by an independent thread, and the generated grasping targets are stored in the task queue for the robotic arm to schedule. When the grasping queue is empty, a new round of photo detection is automatically triggered, realizing the decoupling and asynchronous operation of the observation process and the grasping process.

Images