A load device cooperative control system and method based on visual recognition

Abstract

The application belongs to the technical field of computer vision and intelligent control, and discloses a load device cooperative control system and method based on visual identification, which comprises a data acquisition module, an industrial camera is arranged in a work area, multi-view image acquisition of a load device and a work target is carried out, visual image data is acquired, time sequencing is carried out, and work visual data flow is formed; a pose analysis module is used for image analysis of the work visual data flow, extraction of load device structure characteristics, calculation of the spatial pose of each load device in combination with camera calibration parameters, determination of the work target position, generation of load device pose information; a shielding modeling module is used for equivalent of the geometric shape of each load device to a space bounding box changing with time according to the load device pose information, determination of the position and scale of the bounding box based on preset size parameters, formation of a space shielding body set, and improvement of the efficiency and safety of multi-load device cooperative work.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent control technology, and more specifically, to a collaborative control system and method for a load device based on visual recognition. Background Technology

[0002] Existing load device collaborative control systems and methods mainly suffer from the following problems:

[0003] In industrial automation environments, load cells (such as robotic arms, lifting devices, transport devices, or handling equipment) typically need to work collaboratively within the same work area to complete tasks such as target grasping, handling, or assembly. To achieve real-time monitoring and scheduling of the work process, some systems have begun to introduce visual recognition technology. This involves using industrial cameras to capture images of the work area and employing image recognition algorithms to detect the position of work targets or equipment, thereby assisting in monitoring and controlling the work status. However, existing load cell collaborative control systems still have certain limitations in terms of visual recognition and collaborative control.

[0004] In existing visual monitoring systems, most systems only perform target detection or simple localization for a single device, typically relying on single-view images to identify equipment or work targets, lacking a three-dimensional pose analysis mechanism based on multi-view visual data. When multiple load devices move simultaneously within the same work area, the system struggles to accurately acquire the spatial and orientational relationships between the devices, resulting in a lack of reliable spatial foundation data for the work control system, thus affecting the collaborative operation effect between multiple devices.

[0005] Furthermore, in multi-device collaborative operation environments, the structural components of different load devices are prone to spatial occlusion during movement, such as mutual occlusion between robotic arms, lifting devices, or transportation devices. When the vision system does not perform spatial modeling of the equipment geometry, it is difficult to determine from a spatial structural perspective whether the observation path is occluded by other devices. This can lead to the target being misjudged as unidentifiable or failing to be located in the visual image, affecting the accuracy of the visual recognition results.

[0006] Meanwhile, traditional visual recognition systems typically rely solely on image recognition results to determine the existence of a target, such as using target detection algorithms to determine if a target appears in an image. When a target is temporarily invisible due to occlusion by equipment structure, the visual system often only reports a recognition failure or target loss, without being able to determine whether the invisibility is due to visual occlusion or the target actually leaving the work area. This makes it difficult for the system to accurately analyze the visual recognition results, thereby affecting the stability of system operation.

[0007] On the other hand, in existing technologies, even if the vision system detects that a target is occluded, the system typically only issues an alarm, lacking an active adjustment mechanism based on the occlusion relationship. It cannot automatically adjust the device's posture or movement path to remove the occlusion based on the spatial relationship between devices. This not only reduces the efficiency of multi-load device collaborative operation but may also lead to equipment interference or repetitive adjustments during operation.

[0008] In view of this, the present invention proposes a visual recognition-based load device collaborative control system and method to solve the above problems. Summary of the Invention

[0009] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a load device collaborative control system based on visual recognition, comprising:

[0010] The data acquisition module is used to deploy industrial cameras in the work area to acquire multi-view images of the load device and the work target, obtain visual image data, and perform time-series processing to form a work visual data stream.

[0011] The pose analysis module is used to perform image analysis on the operation visual data stream, extract the structural features of the load device, calculate the spatial pose of each load device in combination with the camera calibration parameters, determine the position of the operation target, and generate the pose information of the load device.

[0012] The occlusion modeling module is used to convert the geometric shape of each load device into a time-varying spatial bounding box based on the pose information of the load device, and to determine the position and scale of the bounding box based on preset size parameters, thus forming a set of spatial occlusion bodies.

[0013] The occlusion determination module is used to construct an observation ray based on the optical center of the industrial camera and the position of the target, and to perform spatial intersection detection based on the set of spatial occlusion bodies to determine whether the observation ray intersects with any spatial bounding box between the industrial camera and the target, so as to determine the visual visibility status of the target.

[0014] The collaborative control module is used to analyze the visual occlusion relationship between load devices based on the visually visible state, generate corresponding load device attitude or motion adjustment commands, and collaboratively adjust the operation status.

[0015] Preferably, the method for acquiring visual image data includes:

[0016] A predetermined number of industrial cameras are installed at different locations in the pre-defined work area. The installation height, pitch angle and horizontal orientation of the industrial cameras are adjusted by the bracket structure so that the field of view of each industrial camera covers the movement area of ​​the load device and the work area where the work target is located, and different observation angles are formed between adjacent industrial cameras.

[0017] During the operation of the industrial camera, images of the load device and the target are continuously acquired through a preset acquisition cycle, and the acquired visual image data is transmitted to the control terminal through a communication interface, thereby obtaining visual image data from different perspectives.

[0018] Preferably, the method for forming a job vision data stream includes:

[0019] Add corresponding timestamp information to the visual image data acquired by each industrial camera, and sort the visual image data according to the acquisition time order; within the same time window, perform time alignment processing on the visual image data from different industrial cameras, and organize them continuously according to the time order to form a time series set, and output the time series set as the job visual data stream.

[0020] Preferably, the method for extracting the structural features of the load device includes:

[0021] Image preprocessing is performed on the visual image data in the task visual data stream. Image preprocessing includes denoising the visual image data using median filtering or Gaussian filtering, and adjusting the brightness using histogram equalization.

[0022] After preprocessing, Canny edge detection or Sobel edge detection methods are used to extract edges from the image to obtain the contour boundary information of the payload device in the visual image data. Corner detection is used to extract key feature points on the contour boundary information. Key feature points include structural corner points, contour curvature extreme points, structural connection points, and contour endpoints. Feature descriptors for corresponding key feature points are calculated, and the similarity between different feature descriptors is calculated using cosine similarity.

[0023] Feature matching is performed between visual image data acquired by different industrial cameras based on the similarity between feature descriptors. Key feature points from images from different perspectives are determined by nearest neighbor search, thereby obtaining key feature points of the same load device under different perspectives and forming the structural features of the load device.

[0024] Preferably, the method for generating the pose information of the load device includes:

[0025] Obtain the camera calibration parameters of each industrial camera, including the camera intrinsic parameter matrix and the camera extrinsic parameter matrix; based on the key feature points matched in the images of different industrial cameras, use triangulation to calculate the spatial coordinates of the corresponding key feature points in three-dimensional space, thereby obtaining the three-dimensional position of the key feature points of the load device.

[0026] Based on the three-dimensional position of the key feature points of the load device, the spatial position parameters and attitude angle parameters of the load device in the global coordinate system are calculated according to the spatial geometric relationship between the key feature points. At the same time, the feature regions of the target in different viewpoint images are located, and the spatial position of the target in the global coordinate system is determined using the same triangulation method, thereby generating load device pose information containing load device position parameters, attitude angle parameters and the spatial position of the target.

[0027] Preferably, the method for obtaining the spatial bounding box includes:

[0028] Obtain the spatial position parameters and attitude angle parameters of each load device in the global coordinate system, and determine the geometric dimension parameters of the load device in the length, width and height directions based on the three-dimensional structural design model of the load device;

[0029] The center position of the bounding box is determined by the spatial position parameters of the load device, and the geometric dimension parameters are used as the scale range of the bounding box in the three coordinate axes. The bounding box is subjected to spatial rotation transformation according to the attitude angle parameters so that the spatial orientation of the bounding box is consistent with the current attitude of the load device, thereby constructing a spatial bounding box in the global coordinate system that corresponds to the spatial occupancy range of the load device.

[0030] Preferably, the method for forming a spatial obstruction assembly includes:

[0031] Obtain the preset dimensional parameters corresponding to each load device. The preset dimensional parameters include the structural dimensions of the load device in the length, width and height directions. Based on the center position of the spatial bounding box and the preset dimensional parameters of each load device, determine the spatial range of the spatial bounding box in the three coordinate axes, thereby determining the position and scale of the bounding box in the global coordinate system.

[0032] The spatial bounding box construction and determination process is performed on all payload devices to obtain the spatial bounding box corresponding to each payload device at the current time. The spatial bounding boxes are then aggregated to form a set of spatial occlusion volumes used to represent potential visual occlusion areas.

[0033] Preferably, the method for determining the visually visible state of the target includes:

[0034] Obtain the optical center position of the industrial camera and the position of the target. Based on the optical center position of the industrial camera and the spatial position of the target, construct an observation ray from the industrial camera to the target, and use the line segment of the observation ray between the optical center of the industrial camera and the target as the visibility detection path.

[0035] Retrieve a pre-constructed set of spatial obstructions, which includes the spatial bounding boxes corresponding to each load device; perform spatial intersection detection on the observation ray and each spatial bounding box in the set of spatial obstructions to determine whether the path of the observation ray between the optical center of the industrial camera and the target enters the spatial boundary range of any spatial bounding box.

[0036] When the observation ray intersects with any spatial bounding box between the optical center of the industrial camera and the target, it is determined that the load device represented by the corresponding spatial bounding box obstructs the observation path, and the target is determined to be visually invisible from the perspective of the corresponding industrial camera.

[0037] When the observed ray does not intersect with any spatial bounding box between the optical center of the industrial camera and the target, it is determined that there is no occlusion relationship between the industrial camera and the target, thus confirming that the target is visually visible from the perspective of the corresponding industrial camera.

[0038] Preferably, the method for collaboratively adjusting the work status includes:

[0039] Based on the visual visibility, identify the load devices that have a spatial intersection relationship with the target on the propagation path of the observed ray, and record the position of the load device that causes occlusion, the position of the occluded target, and the identification of the observed industrial camera to construct a set of visual occlusion relationships.

[0040] Based on the position of the load device that causes the obstruction, the position of the obstructed target, and the position of the optical center of the industrial camera, calculate the spatial offset direction of the load device that causes the obstruction relative to the observation ray, and determine the adjustment direction that can make the spatial bounding box deviate from the propagation path of the observation ray.

[0041] Based on the adjustment direction, the corresponding load device attitude adjustment amount or spatial displacement amount is generated, forming a load device attitude or motion adjustment command; the attitude or motion adjustment command is sent to the corresponding load device control terminal, so that the obstructing load device changes its attitude or spatial position according to the attitude or motion command, realizing the coordinated operation adjustment between load devices.

[0042] A visual recognition-based collaborative control method for load devices includes:

[0043] S1. Deploy industrial cameras in the work area to acquire multi-view images of the load device and the work target, obtain visual image data, and perform time-series processing to form a work visual data stream.

[0044] S2. Perform image analysis on the visual data stream of the operation, extract the structural features of the load device, calculate the spatial pose of each load device in combination with the camera calibration parameters, determine the position of the operation target, and generate the pose information of the load device.

[0045] S3. Based on the position and orientation information of the load devices, the geometric shape of each load device is equivalent to a spatial bounding box that changes over time, and the position and size of the bounding box are determined based on preset size parameters to form a set of spatial occlusion bodies.

[0046] S4. Construct an observation ray based on the optical center of the industrial camera and the position of the target, and perform spatial intersection detection based on the set of spatial occlusions to determine whether the observation ray intersects with any spatial bounding box between the industrial camera and the target, so as to determine the visual visibility status of the target.

[0047] S5. Analyze the visual occlusion relationship between load devices based on the visually visible state, generate corresponding load device attitude or motion adjustment commands, and coordinate the adjustment of the operation status.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] By acquiring multi-view visual images using multiple industrial cameras, and calculating the 3D spatial coordinates of key feature points of the load devices using key feature point matching and triangulation, the position and attitude angle parameters of these points in the global coordinate system are obtained, enabling high-precision spatial pose analysis of multiple load devices. Based on the 3D structural design model of the load devices, it is equivalent to a time-varying spatial bounding box. Combined with real-time pose information, a set of spatial occupancy bodies is constructed to dynamically describe the spatial occupancy range of each load device, achieving real-time modeling of potential visual occupancy areas and solving the problem that traditional vision systems cannot describe the spatial structure of equipment.

[0050] A ray is constructed between the optical center of an industrial camera and the target object. Spatial intersection detection is performed between this ray and the bounding box to determine if the observation path is obstructed. This enables geometric determination of the target's visual visibility, analyzing visual occlusion relationships from a spatial geometric perspective and improving the stability and reliability of the visual recognition system. The spatial offset direction of the occlusion device relative to the ray is calculated, generating adjustment commands to adjust the device's position or orientation. This achieves linkage between visual perception and equipment control, automatically avoiding visual occlusion problems and improving the efficiency and stability of multi-device collaborative operations. In complex environments where multiple load devices operate simultaneously, real-time analysis of equipment spatial relationships and automatic adjustment of equipment posture or movement paths reduce visual recognition failures and improve overall operational efficiency and safety.

[0051] Compared to traditional methods relying on image recognition results, this invention directly determines visual occlusion relationships based on the three-dimensional spatial structure, improving the accuracy of target visibility determination. It retrieves a set of spatial occlusion bodies and performs intersection detection between the observation ray and the spatial bounding box to determine if any payload device is located on the observation path, identifying dynamic visual occlusion relationships in real time and improving the reliability of visual monitoring. When a target is temporarily invisible in the visual image, the spatial relationship between the observation ray and the spatial bounding box is used to determine the reason for invisibility, reducing visual recognition errors and improving the stability of the vision system. Accurately determining the target's visual visibility status provides reliable data for the collaborative control module, enabling optimized collaborative operation of payload devices. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of a load device collaborative control system based on visual recognition according to the present invention.

[0053] Figure 2 This is a schematic diagram of a collaborative control method for a load device based on visual recognition according to the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Example 1:

[0056] Please see Figure 1 As shown, this embodiment provides a visual recognition-based load device collaborative control system, which specifically includes the following steps:

[0057] The data acquisition module is used to deploy industrial cameras in the work area to acquire multi-view images of the load device and the work target, obtain visual image data, and perform time-series processing to form a work visual data stream.

[0058] The pose analysis module is used to perform image analysis on the operation visual data stream, extract the structural features of the load device, calculate the spatial pose of each load device in combination with the camera calibration parameters, determine the position of the operation target, and generate the pose information of the load device.

[0059] The occlusion modeling module is used to convert the geometric shape of each load device into a time-varying spatial bounding box based on the pose information of the load device, and to determine the position and scale of the bounding box based on preset size parameters, thus forming a set of spatial occlusion bodies.

[0060] The occlusion determination module is used to construct an observation ray based on the optical center of the industrial camera and the position of the target, and to perform spatial intersection detection based on the set of spatial occlusion bodies to determine whether the observation ray intersects with any spatial bounding box between the industrial camera and the target, so as to determine the visual visibility status of the target.

[0061] The collaborative control module is used to analyze the visual occlusion relationship between load devices based on the visually visible state, generate corresponding load device attitude or motion adjustment commands, and collaboratively adjust the operation status.

[0062] Methods for acquiring visual image data include:

[0063] A predetermined number of industrial cameras are installed at different locations in the pre-defined work area. The installation height, pitch angle and horizontal orientation of the industrial cameras are adjusted by the bracket structure so that the field of view of each industrial camera covers the movement area of ​​the load device and the work area where the work target is located, and different observation angles are formed between adjacent industrial cameras.

[0064] It should be noted that in this embodiment, the visual monitoring range is first determined based on the maximum working range of the load device and the spatial distribution of the target, and then extended outward by a preset safety boundary distance, which includes both the movement trajectory of the load device and the position of the target, thereby forming a preset working area for visual acquisition.

[0065] The preset work area is a three-dimensional spatial area covering the movement range of the load device and the location of the work target. Then, the number of industrial cameras is determined according to the spatial size of the preset work area and the visual coverage requirements. When the preset work area is a small to medium-sized work space, 3-6 industrial cameras can be deployed around the work area. Four industrial cameras are deployed on the front, sides and top of the work area to form a multi-directional observation structure. The installation height, pitch angle and horizontal orientation of the industrial cameras are adjusted by the bracket structure so that the field of view of each industrial camera can cover the movement area of ​​the load device and the area where the work target is located. At the same time, it is ensured that different observation angles are formed between adjacent industrial cameras to obtain multi-view image information of the work area.

[0066] During the operation of the industrial camera, images of the load device and the target are continuously acquired through a preset acquisition cycle, and the acquired visual image data is transmitted to the control terminal through a communication interface, thereby obtaining visual image data from different perspectives.

[0067] Methods for generating job vision data streams include:

[0068] Add corresponding timestamp information to the visual image data acquired by each industrial camera, and sort the visual image data according to the acquisition time order; within the same time window, perform time alignment processing on the visual image data from different industrial cameras, and organize them continuously according to the time order to form a time series set, and output the time series set as the job visual data stream.

[0069] Methods for extracting structural features of load devices include:

[0070] Image preprocessing is performed on the visual image data in the task visual data stream. Image preprocessing includes denoising the visual image data using median filtering or Gaussian filtering, and adjusting the brightness using histogram equalization.

[0071] After preprocessing, Canny edge detection or Sobel edge detection methods are used to extract edges from the image to obtain the contour boundary information of the payload device in the visual image data. Corner detection is used to extract key feature points on the contour boundary information. Key feature points include structural corner points, contour curvature extreme points, structural connection points, and contour endpoints. Feature descriptors for corresponding key feature points are calculated, and the similarity between different feature descriptors is calculated using cosine similarity.

[0072] It should be noted that key feature points correspond to the edges where they intersect, locations of significant curvature changes, or structural connections within the geometry of the load device, and are used to characterize the stable geometric features of the load device. After detecting key feature points, the feature descriptor can be calculated using the following steps:

[0073] First, a fixed-size local neighborhood window, such as a 16×16 or 32×32 pixel region, is extracted from the visual image, centered on the key feature points. This region can be understood as a small "image fingerprint region" surrounding the feature points. Next, the grayscale gradient of each pixel is calculated within this local neighborhood window. The gradient can be obtained using the Sobel operator, including both horizontal and vertical gradients.

[0074] The gradient magnitude and direction are calculated based on the horizontal and vertical gradients. The local neighborhood window is then divided into multiple sub-regions, such as 4×4 sub-blocks. Within each sub-region, the gradient distribution in different directions is statistically analyzed; for example, the gradient direction is divided into eight directional intervals, and the cumulative gradient intensity in each direction is calculated to obtain the gradient direction histogram for that sub-region.

[0075] Finally, the gradient direction histograms of all sub-regions are sequentially concatenated to form a feature vector, which is the feature descriptor of the key feature point. For example, 4×4×8 directions = 128-dimensional feature vector. This 128-dimensional feature vector essentially describes the spatial distribution characteristics of the image texture structure around the key feature point. When calculating the feature descriptor of the key feature point, local neighborhood image patches are extracted with the key feature point as the center, the gray-level gradient of the neighboring pixels is calculated, and the gradient distribution in different directions is statistically analyzed to form a feature vector. After normalizing the feature vector, the similarity between different feature descriptors is obtained by calculating cosine similarity.

[0076] Feature matching is performed between visual image data acquired by different industrial cameras based on the similarity between feature descriptors. Key feature points from images from different perspectives are determined by nearest neighbor search, thereby obtaining key feature points of the same load device under different perspectives and forming the structural features of the load device.

[0077] Methods for generating payload device pose information include:

[0078] Obtain the camera calibration parameters of each industrial camera, including the camera intrinsic parameter matrix and the camera extrinsic parameter matrix; based on the key feature points matched in the images of different industrial cameras, use triangulation to calculate the spatial coordinates of the corresponding key feature points in three-dimensional space, thereby obtaining the three-dimensional position of the key feature points of the load device.

[0079] It should be noted that the camera intrinsic parameter matrix describes imaging parameters such as camera focal length, principal point position, and pixel scale, while the camera extrinsic parameter matrix describes the spatial position and orientation of the camera coordinate system relative to the global coordinate system. For the pixel coordinates of the same key feature point in images from different industrial cameras, the pixel coordinates are converted into normalized imaging coordinates in the camera coordinate system using the intrinsic parameter matrix of the corresponding industrial camera, and the spatial observation ray direction of the key feature point in each camera coordinate system is determined based on the extrinsic parameter matrix.

[0080] Since different industrial cameras observe the same key feature point from different spatial locations, the corresponding observation rays in each camera's coordinate system will intersect in space. Based on the geometric relationships between these observation rays, the system calculates the intersection point or minimum error intersection point of each observation ray in space using triangulation methods, thereby obtaining the three-dimensional spatial coordinates of the key feature point in the global coordinate system. By performing the above triangulation calculations on multiple key feature points, the set of three-dimensional positions of each key feature point of the load device in the global coordinate system can be obtained.

[0081] Based on the three-dimensional position of the key feature points of the load device, the spatial position parameters and attitude angle parameters of the load device in the global coordinate system are calculated according to the spatial geometric relationship between the key feature points. At the same time, the feature regions of the target in different viewpoint images are located, and the spatial position of the target in the global coordinate system is determined using the same triangulation method, thereby generating load device pose information containing load device position parameters, attitude angle parameters and the spatial position of the target.

[0082] It should be noted that after obtaining the three-dimensional spatial coordinates of multiple key feature points of the load device, the system calculates the spatial position parameters and attitude angle parameters of the load device in the global coordinate system based on the spatial geometric relationships between these key feature points. Specifically, the structural feature points of the load device are treated as a set of rigid body structural feature points, and the theoretical positions of each key feature point in the local coordinate system of the load device are determined based on the three-dimensional structural model of the load device. Specifically, during the load device design phase, several feature positions with stable geometric structures are selected as key feature points in the three-dimensional structural design model of the load device, such as structural contour corners, structural connection points, or edge endpoints of the load device. The three-dimensional structural design model can be a three-dimensional CAD model or a digital structural model of the load device formed during the product design phase.

[0083] Subsequently, a local coordinate system for the load device is established using the structural center or a preset installation reference point as the origin. The spatial positions of key feature points in the structural model are then converted into three-dimensional coordinates relative to this local coordinate system, thus obtaining the theoretical positions of each key feature point in the local coordinate system. In subsequent pose calculations, the three-dimensional coordinates of the key feature points obtained through triangulation are spatially correlated with the aforementioned theoretical positions to calculate the spatial transformation relationship between the local coordinate system and the global coordinate system of the load device.

[0084] Using the three-dimensional coordinates of key feature points obtained by triangulation as observation points, the spatial transformation relationship between the local coordinate system and the global coordinate system is calculated through the spatial rigid body transformation solution method. The spatial transformation relationship includes a three-dimensional translation vector and a three-dimensional rotation matrix, where the three-dimensional translation vector represents the spatial position parameters of the load device in the global coordinate system, and the three-dimensional rotation matrix is ​​used to describe the spatial attitude of the load device. Furthermore, the attitude angle parameters of the load device, including roll angle, pitch angle and yaw angle, are calculated based on the rotation matrix, thereby obtaining the complete pose information of the load device in the global coordinate system.

[0085] While determining the pose of the load device, the system also locates the feature regions of the target in visual image data acquired by different industrial cameras. Specifically, the target region is identified in each industrial camera image, and the center point or boundary feature points of the target region are extracted as target feature points. Subsequently, the target feature points are matched between images from different viewpoints to determine the corresponding target feature points from different industrial camera images. After obtaining the corresponding target feature points, the system uses the same triangulation method as for the 3D reconstruction of key feature points to calculate the intersection relationship of the corresponding observation rays in space based on the camera calibration parameters of different industrial cameras and the pixel coordinates of the corresponding target feature points, thereby determining the 3D spatial position of the target in the global coordinate system.

[0086] Methods for obtaining spatial bounding boxes include:

[0087] Obtain the spatial position parameters and attitude angle parameters of each load device in the global coordinate system, and determine the geometric dimension parameters of the load device in the length, width and height directions based on the three-dimensional structural design model of the load device;

[0088] It should be noted that, in this embodiment, in order to obtain the geometric dimension parameters of the load device for spatial modeling, a three-dimensional structural design model of the load device is first obtained. Then, a local coordinate system consistent with the installation direction of the load device is established in the three-dimensional structural design model, and the three orthogonal axes of the local coordinate system correspond to the length direction, width direction and height direction of the load device, respectively. Spatial boundary analysis is performed on the three-dimensional structural model under this coordinate system, and the maximum and minimum coordinate values ​​of the model in each coordinate axis direction are extracted. By calculating the difference between the maximum and minimum coordinate values ​​in the corresponding coordinate axis directions, the geometric dimension parameters of the load device in the length direction, width direction and height direction are obtained respectively.

[0089] The center position of the bounding box is determined by the spatial position parameters of the load device, and the geometric dimension parameters are used as the scale range of the bounding box in the three coordinate axes. The bounding box is subjected to spatial rotation transformation according to the attitude angle parameters so that the spatial orientation of the bounding box is consistent with the current attitude of the load device, thereby constructing a spatial bounding box in the global coordinate system that corresponds to the spatial occupancy range of the load device.

[0090] The spatial bounding box is: ;in, Indicates the first Each load device at time The corresponding bounding box; Indicates the number index of the load device; Representing three-dimensional space any point in it; Represents three-dimensional space; Indicates the first The spatial enclosure of each load device is indicated at the center position; Point Between the centers of the spatial bounding box, at the first The distance difference along each coordinate axis, for example, when When, it means Difference in directional distance; Indicates the first The load device in the first Geometric dimension parameters along each coordinate axis, for example Indicates the length dimension of the load device. Indicates the width dimension of the load-bearing device; The constraint condition must be true in three directions simultaneously; a point is considered valid only when all three conditions are met. Only then does it belong to the enclosing box; Indicates the index of the coordinate axis direction;

[0091] This solution addresses the following technical problems in existing technologies: Most current visual monitoring systems only perform target detection or simple localization on a single device, lacking a 3D pose analysis mechanism based on multi-view visual data. When multiple load devices move simultaneously within the same work area, the system struggles to accurately acquire the spatial and orientational relationships between the devices, resulting in a lack of reliable spatial foundation data for subsequent operational coordination. In multi-device collaborative environments, structural occlusion easily occurs between load devices, such as mutual occlusion between robotic arms, lifting devices, or transport devices. When the vision system does not spatially model the device geometry, it cannot determine whether the visual observation path is obstructed by other devices, leading to the target being misjudged as unidentifiable or failing to be located. Traditional vision systems typically rely solely on image recognition results to determine target presence, lacking a visibility calculation mechanism based on spatial geometric relationships. When a target is temporarily invisible due to occlusion, the system struggles to determine whether the actual cause is visual occlusion or target loss, thus affecting system stability. In existing technologies, even if visual occlusion is detected, the system usually only issues an alarm, but cannot actively adjust the equipment's posture or movement path based on the occlusion relationship, resulting in low collaborative operation efficiency and easy to cause operation conflicts or repeated adjustments.

[0092] Compared to existing technologies, the advantages are as follows: By acquiring multi-view visual images using multiple industrial cameras and calculating the 3D spatial coordinates of key feature points of the load device using key feature point matching and triangulation methods, the position and attitude angle parameters of the load device in the global coordinate system are obtained, achieving high-precision spatial pose analysis of multiple load devices. Based on the 3D structural design model of the load device, each load device is equivalent to a time-varying spatial bounding box, and a set of spatial occlusion bodies is constructed by combining the real-time pose information of the equipment. This allows for dynamic description of the spatial occupancy range of each load device in 3D space, thereby achieving real-time modeling of potential visual occlusion areas and solving the problem that traditional vision systems cannot describe the spatial structure of equipment. By constructing an observation ray between the optical center of the industrial camera and the target, and performing spatial intersection detection on the observation ray and the spatial bounding box, it is possible to determine whether the observation path is occluded by the load device, achieving geometric determination of the visual visibility status of the target. It can analyze visual occlusion relationships from a spatial geometric perspective, enabling the system to accurately determine the cause of invisibility when the target is occluded, thereby improving the stability and reliability of the visual recognition system.

[0093] By calculating the spatial offset direction of the occlusion device relative to the observed ray and generating corresponding attitude or motion adjustment commands, the occlusion device can be adjusted in position or attitude along the direction where the occlusion can be removed. This enables the linkage between visual perception and equipment control, allowing the load devices to automatically avoid visual occlusion problems during collaborative operations, improving the efficiency and stability of multi-device collaborative operations. It can analyze the spatial relationships between devices in real time in complex environments where multiple load devices operate simultaneously, and automatically adjust the device attitude or motion path, thereby reducing problems such as visual recognition failure, equipment conflicts, and operational stagnation, and improving overall operational efficiency and safety.

[0094] Methods for forming a spatial occlusion set include:

[0095] Obtain the preset dimensional parameters corresponding to each load device. The preset dimensional parameters include the structural dimensions of the load device in the length, width and height directions. Based on the center position of the spatial bounding box and the preset dimensional parameters of each load device, determine the spatial range of the spatial bounding box in the three coordinate axes, thereby determining the position and scale of the bounding box in the global coordinate system.

[0096] It should be noted that, in this embodiment, after obtaining the center position of the spatial bounding box and the preset size parameters of each load device, the spatial range of the spatial bounding box in the global coordinate system is determined. Specifically, the center position of the spatial bounding box corresponding to the load device is taken as the geometric center point of the bounding box, and the length, width, and height dimensions in the preset size parameters are mapped to the three orthogonal coordinate axes of the global coordinate system to characterize the structural scale of the bounding box in each direction. Then, in each coordinate axis direction, based on the center position, half of the corresponding structural dimension is extended in the positive and negative directions respectively, thereby determining the minimum and maximum boundary positions of the bounding box in that coordinate axis direction. By calculating the minimum and maximum boundary coordinate values ​​in the three coordinate axis directions respectively, the boundary range of the spatial bounding box in three-dimensional space is formed, thereby determining the position and scale of the corresponding load device's spatial bounding box in the global coordinate system, so that the bounding box can characterize the spatial range occupied by the load device at the current moment.

[0097] The spatial bounding box construction and determination process is performed on all payload devices to obtain the spatial bounding box corresponding to each payload device at the current time. The spatial bounding boxes are then aggregated to form a set of spatial occlusion volumes used to represent potential visual occlusion areas.

[0098] Methods for determining the visual visibility state of a target include:

[0099] Obtain the optical center position of the industrial camera and the position of the target. Based on the optical center position of the industrial camera and the spatial position of the target, construct an observation ray from the industrial camera to the target, and use the line segment of the observation ray between the optical center of the industrial camera and the target as the visibility detection path.

[0100] Observation rays: ;in, Indicates the first The first industrial camera observation The observation rays formed when each task target is identified; Indicates the first The optical center position of an industrial camera; Indicates the first The spatial position of each task target in the global coordinate system; This represents the ray parameters, used to control the position of points on the ray;

[0101] Retrieve a pre-constructed set of spatial obstructions, which includes the spatial bounding boxes corresponding to each load device; perform spatial intersection detection on the observation ray and each spatial bounding box in the set of spatial obstructions to determine whether the path of the observation ray between the optical center of the industrial camera and the target enters the spatial boundary range of any spatial bounding box.

[0102] When the observed ray intersects with any spatial bounding box between the optical center of the industrial camera and the target (i.e.) It determines that the load device represented by the corresponding spatial bounding box obstructs the observation path and determines that the target is visually invisible from the perspective of the corresponding industrial camera.

[0103] When the observed ray does not intersect with any spatial bounding box between the optical center of the industrial camera and the target, it is determined that there is no occlusion relationship between the industrial camera and the target, thus confirming that the target is visually visible from the perspective of the corresponding industrial camera.

[0104] This solution addresses the following technical problems in existing technologies: In traditional visual recognition systems, target visibility is typically determined solely by image recognition results, such as using target detection algorithms to determine the presence of a target in an image. When a target is occluded by other devices or structures, the vision system often only reports a recognition failure or target loss, failing to determine the specific cause of the failure at the spatial structure level. This results in the system's inability to distinguish between actual target disappearance and visual occlusion. In collaborative environments with multiple payload devices, the trajectories and postures of different devices in space are constantly changing, easily leading to dynamic occlusion relationships between device structures. Existing vision systems typically lack the ability to model the spatial structure of devices, failing to determine whether a device is on the observation path based on its 3D spatial position and geometry, thus making it difficult to accurately identify visual occlusion relationships between devices. Traditional vision methods rely on passive image recognition; the system only detects recognition failure after the target is occluded, lacking a visibility prediction mechanism based on spatial geometric relationships. Therefore, it cannot predict in advance whether an observation path might be occluded by a device structure, resulting in poor stability of visual recognition.

[0105] Advantages compared to existing technologies: Compared to traditional methods relying on image recognition results, this invention can directly determine visual occlusion relationships based on three-dimensional spatial structures, thereby improving the accuracy of target visibility determination. By retrieving a set of spatial occlusion bodies and performing spatial intersection detection on the observation ray and each spatial bounding box, it can determine whether a payload device is located on the observation path. When the observation ray intersects with a spatial bounding box, it can be determined that the corresponding payload device is occluding the observation path. Through this mechanism, the system can identify dynamic visual occlusion relationships between devices in real time in a multi-payload collaborative operation environment, improving the reliability of visual monitoring in complex operating environments. When a target is temporarily invisible in the visual image, this invention can determine whether the reason for the target's invisibility is device occlusion by the spatial relationship between the observation ray and the spatial bounding box, rather than simply considering it as a lost target, thereby reducing visual recognition misjudgments and improving the stability of the vision system. By accurately determining the target's visual visibility state, this invention can provide reliable spatial occlusion relationship data for subsequent collaborative control modules, enabling the system to generate corresponding posture or motion adjustment commands based on the occlusion relationship, thereby optimizing the collaborative operation between payload devices.

[0106] Methods for collaboratively adjusting work status include:

[0107] Based on the visual visibility, identify the load devices that have a spatial intersection relationship with the target on the propagation path of the observed ray, and record the position of the load device that causes occlusion, the position of the occluded target, and the identification of the observed industrial camera to construct a set of visual occlusion relationships.

[0108] Based on the position of the load device that causes the obstruction, the position of the obstructed target, and the position of the optical center of the industrial camera, calculate the spatial offset direction of the load device that causes the obstruction relative to the observation ray, and determine the adjustment direction that can make the spatial bounding box deviate from the propagation path of the observation ray.

[0109] It should be noted that after identifying the load device causing the obstruction, the spatial relationship between the observation ray and the obstructing device is determined based on the position of the industrial camera's optical center, the position of the obstructed target, and the position of the obstructing load device. First, the line connecting the industrial camera's optical center and the target position is taken as the direction of the observation ray. A spatial vector is constructed based on the positions of the industrial camera's optical center and the center of the obstructing load device, and this spatial vector is projected onto the direction of the observation ray to obtain the projection component of the obstructing load device in the direction of the observation ray. Then, the projection component is subtracted from the spatial vector of the obstructing load device to obtain the lateral offset vector of the obstructing load device relative to the observation ray. The direction of this lateral offset vector is the spatial offset direction of the obstructing load device relative to the observation ray, and this offset direction is used as the reference direction for adjusting the attitude or position of the load device.

[0110] Based on the adjustment direction, the corresponding load device attitude adjustment amount or spatial displacement amount is generated, forming a load device attitude or motion adjustment command; the attitude or motion adjustment command is sent to the corresponding load device control terminal, so that the obstructing load device changes its attitude or spatial position according to the attitude or motion command, realizing the coordinated operation adjustment between load devices.

[0111] In this embodiment, multiple industrial cameras are used to acquire multi-view visual images. Key feature point matching and triangulation are used to calculate the three-dimensional spatial coordinates of key feature points of the load device, obtaining its position and attitude angle parameters in the global coordinate system, thus achieving high-precision spatial pose analysis of multiple load devices. Based on the three-dimensional structural design model of the load device, it is equivalent to a time-varying spatial bounding box. Combined with real-time pose information, a set of spatial occupancy bodies is constructed to dynamically describe the spatial occupancy range of each load device, achieving real-time modeling of potential visual occupancy areas and solving the problem that traditional vision systems cannot describe the spatial structure of equipment.

[0112] A ray is constructed between the optical center of an industrial camera and the target object. Spatial intersection detection is performed between this ray and the bounding box to determine if the observation path is obstructed. This enables geometric determination of the target's visual visibility, analyzing visual occlusion relationships from a spatial geometric perspective and improving the stability and reliability of the visual recognition system. The spatial offset direction of the occlusion device relative to the ray is calculated, generating adjustment commands to adjust the device's position or orientation. This achieves linkage between visual perception and equipment control, automatically avoiding visual occlusion problems and improving the efficiency and stability of multi-device collaborative operations. In complex environments where multiple load devices operate simultaneously, real-time analysis of equipment spatial relationships and automatic adjustment of equipment posture or movement paths reduce visual recognition failures and improve overall operational efficiency and safety.

[0113] Compared to traditional methods relying on image recognition results, this invention directly determines visual occlusion relationships based on the three-dimensional spatial structure, improving the accuracy of target visibility determination. It retrieves a set of spatial occlusion bodies and performs intersection detection between the observation ray and the spatial bounding box to determine if any payload device is located on the observation path, identifying dynamic visual occlusion relationships in real time and improving the reliability of visual monitoring. When a target is temporarily invisible in the visual image, the spatial relationship between the observation ray and the spatial bounding box is used to determine the reason for invisibility, reducing visual recognition errors and improving the stability of the vision system. Accurately determining the target's visual visibility status provides reliable data for the collaborative control module, enabling optimized collaborative operation of payload devices.

[0114] Example 2:

[0115] Please see Figure 2 As shown, parts not described in detail in this embodiment are described in Embodiment 1. A visual recognition-based collaborative control method for a load device is provided, including:

[0116] S1. Deploy industrial cameras in the work area to acquire multi-view images of the load device and the work target, obtain visual image data, and perform time-series processing to form a work visual data stream.

[0117] S2. Perform image analysis on the visual data stream of the operation, extract the structural features of the load device, calculate the spatial pose of each load device in combination with the camera calibration parameters, determine the position of the operation target, and generate the pose information of the load device.

[0118] S3. Based on the position and orientation information of the load devices, the geometric shape of each load device is equivalent to a spatial bounding box that changes over time, and the position and size of the bounding box are determined based on preset size parameters to form a set of spatial occlusion bodies.

[0119] S4. Construct an observation ray based on the optical center of the industrial camera and the position of the target, and perform spatial intersection detection based on the set of spatial occlusions to determine whether the observation ray intersects with any spatial bounding box between the industrial camera and the target, so as to determine the visual visibility status of the target.

[0120] S5. Analyze the visual occlusion relationship between load devices based on the visually visible state, generate corresponding load device attitude or motion adjustment commands, and coordinate the adjustment of the operation status.

[0121] Since the electronic device described in this embodiment is the electronic device used to implement the visual recognition-based load device collaborative control system and method described in this application embodiment, those skilled in the art can understand the specific implementation methods and various variations of the electronic device in this embodiment based on the visual recognition-based load device collaborative control system and method described in this application embodiment. Therefore, how the electronic device implements the method in this application embodiment will not be described in detail here. Any electronic device used by those skilled in the art to implement the visual recognition-based load device collaborative control system and method described in this application embodiment falls within the protection scope of this application.

[0122] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0123] The above description is merely a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for users of ordinary technical skills, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A load device collaborative control system based on vision recognition, characterized in that, include: The data acquisition module is used to deploy industrial cameras in the work area to acquire multi-view images of the load device and the work target, obtain visual image data, and perform time-series processing to form a work visual data stream. The pose analysis module is used to perform image analysis on the operation visual data stream, extract the structural features of the load device, calculate the spatial pose of each load device in combination with the camera calibration parameters, determine the position of the operation target, and generate the pose information of the load device. The occlusion modeling module is used to convert the geometric shape of each load device into a time-varying spatial bounding box based on the pose information of the load device, and to determine the position and scale of the bounding box based on preset size parameters, thus forming a set of spatial occlusion bodies. The method for obtaining the spatial bounding box includes: Obtain the spatial position parameters and attitude angle parameters of each load device in the global coordinate system, and determine the geometric dimension parameters of the load device in the length, width and height directions based on the three-dimensional structural design model of the load device; The center position of the bounding box is determined by the spatial position parameters of the load device, and the geometric dimension parameters are used as the scale range of the bounding box in the three coordinate axis directions. The bounding box is subjected to spatial rotation transformation according to the attitude angle parameters so that the spatial orientation of the bounding box is consistent with the current attitude of the load device, thereby constructing a spatial bounding box corresponding to the spatial occupancy range of the load device in the global coordinate system. The method for forming a spatial obstruction assembly includes: Obtain the preset dimensional parameters corresponding to each load device. The preset dimensional parameters include the structural dimensions of the load device in the length, width and height directions. Based on the center position of the spatial bounding box and the preset dimensional parameters of each load device, determine the spatial range of the spatial bounding box in the three coordinate axes, thereby determining the position and scale of the bounding box in the global coordinate system. Perform the spatial bounding box construction and determination process on all payload devices to obtain the spatial bounding box corresponding to each payload device at the current time, and collect the spatial bounding boxes to form a set of spatial occlusion volumes for representing potential visual occlusion areas; The occlusion determination module is used to construct an observation ray based on the optical center of the industrial camera and the position of the target, and to perform spatial intersection detection based on the set of spatial occlusion bodies to determine whether the observation ray intersects with any spatial bounding box between the industrial camera and the target, so as to determine the visual visibility status of the target. The method for determining the visually visible state of a target includes: Obtain the optical center position of the industrial camera and the position of the target. Based on the optical center position of the industrial camera and the spatial position of the target, construct an observation ray from the industrial camera to the target, and use the line segment of the observation ray between the optical center of the industrial camera and the target as the visibility detection path. Retrieve a pre-constructed set of spatial obstructions, which includes the spatial bounding boxes corresponding to each load device; perform spatial intersection detection on the observation ray and each spatial bounding box in the set of spatial obstructions to determine whether the path of the observation ray between the optical center of the industrial camera and the target enters the spatial boundary range of any spatial bounding box. When the observation ray intersects with any spatial bounding box between the optical center of the industrial camera and the target, it is determined that the load device represented by the corresponding spatial bounding box obstructs the observation path, and the target is determined to be visually invisible from the perspective of the corresponding industrial camera. When the observed ray does not intersect with any spatial bounding box between the optical center of the industrial camera and the target, it is determined that there is no occlusion relationship between the industrial camera and the target, thus determining that the target is visually visible from the perspective of the corresponding industrial camera. The collaborative control module is used to analyze the visual occlusion relationship between load devices based on the visually visible state, generate corresponding load device attitude or motion adjustment commands, and collaboratively adjust the operation status.

2. The load device collaborative control system based on vision recognition according to claim 1, characterized in that, The method for acquiring visual image data includes: A predetermined number of industrial cameras are installed at different locations in the predetermined work area. The installation height, pitch angle and horizontal orientation of the industrial cameras are adjusted by the bracket structure so that the field of view of each industrial camera covers the movement area of ​​the load device and the work area where the work target is located, and different observation angles are formed between adjacent industrial cameras. During the operation of the industrial camera, images of the load device and the target are continuously acquired through a preset acquisition cycle, and the acquired visual image data is transmitted to the control terminal through a communication interface, thereby obtaining visual image data from different perspectives.

3. The load device collaborative control system based on vision recognition according to claim 2, characterized in that, The method for forming the job visual data stream includes: Add corresponding timestamp information to the visual image data acquired by each industrial camera, and sort the visual image data according to the acquisition time order; within the same time window, perform time alignment processing on the visual image data from different industrial cameras, and organize them continuously according to the time order to form a time series set, and output the time series set as the job visual data stream.

4. The load device collaborative control system based on vision recognition according to claim 3, characterized in that, The method for extracting structural features of the load device includes: Image preprocessing is performed on the visual image data in the task visual data stream. Image preprocessing includes denoising the visual image data using median filtering or Gaussian filtering, and adjusting the brightness using histogram equalization. After preprocessing, Canny edge detection or Sobel edge detection methods are used to extract edges from the image to obtain the contour boundary information of the payload device in the visual image data. Corner detection is used to extract key feature points on the contour boundary information. Key feature points include structural corner points, contour curvature extreme points, structural connection points, and contour endpoints. Feature descriptors for corresponding key feature points are calculated, and the similarity between different feature descriptors is calculated using cosine similarity. Feature matching is performed between visual image data acquired by different industrial cameras based on the similarity between feature descriptors. Key feature points from images from different perspectives are determined by nearest neighbor search, thereby obtaining key feature points of the same load device under different perspectives and forming the structural features of the load device.

5. The load device collaborative control system based on vision recognition according to claim 4, characterized in that, The method for generating the pose information of the load device includes: Obtain the camera calibration parameters of each industrial camera, including the camera intrinsic parameter matrix and the camera extrinsic parameter matrix; based on the key feature points matched in the images of different industrial cameras, use triangulation to calculate the spatial coordinates of the corresponding key feature points in three-dimensional space, thereby obtaining the three-dimensional position of the key feature points of the load device. Based on the three-dimensional position of the key feature points of the load device, the spatial position parameters and attitude angle parameters of the load device in the global coordinate system are calculated according to the spatial geometric relationship between the key feature points. At the same time, the feature regions of the target in different viewpoint images are located, and the spatial position of the target in the global coordinate system is determined using the same triangulation method, thereby generating load device pose information containing load device position parameters, attitude angle parameters and the spatial position of the target.

6. The load device collaborative control system based on vision recognition according to claim 5, characterized in that, The method for collaboratively adjusting the work status includes: Based on the visual visibility, identify the load devices that have a spatial intersection relationship with the target on the propagation path of the observed ray, and record the position of the load device that causes occlusion, the position of the occluded target, and the identification of the observed industrial camera to construct a set of visual occlusion relationships. Based on the position of the load device that causes the obstruction, the position of the obstructed target, and the position of the optical center of the industrial camera, calculate the spatial offset direction of the load device that causes the obstruction relative to the observation ray, and determine the adjustment direction that can make the spatial bounding box deviate from the propagation path of the observation ray. Based on the adjustment direction, the corresponding load device attitude adjustment amount or spatial displacement amount is generated, forming a load device attitude or motion adjustment command; the attitude or motion adjustment command is sent to the corresponding load device control terminal, so that the obstructing load device changes its attitude or spatial position according to the attitude or motion command, realizing the coordinated operation adjustment between load devices.

7. A load device collaborative control method based on vision recognition, implemented by a load device collaborative control system based on vision recognition as described in any one of claims 1 to 6, characterized in that, include: S1. Deploy industrial cameras in the work area to acquire multi-view images of the load device and the work target, obtain visual image data, and perform time-series processing to form a work visual data stream. S2. Perform image analysis on the visual data stream of the operation, extract the structural features of the load device, calculate the spatial pose of each load device in combination with the camera calibration parameters, determine the position of the operation target, and generate the pose information of the load device. S3. Based on the position and orientation information of the load devices, the geometric shape of each load device is equivalent to a spatial bounding box that changes over time, and the position and size of the bounding box are determined based on preset size parameters to form a set of spatial occlusion bodies. S4. Construct an observation ray based on the optical center of the industrial camera and the position of the target, and perform spatial intersection detection based on the set of spatial occlusions to determine whether the observation ray intersects with any spatial bounding box between the industrial camera and the target, so as to determine the visual visibility status of the target. S5. Analyze the visual occlusion relationship between load devices based on the visually visible state, generate corresponding load device attitude or motion adjustment commands, and coordinate the adjustment of the operation status.

Images