Industrial robot control method and device based on vision technology

Abstract

The invention relates to the technical field of robot control, in particular to an industrial robot control method and device based on the visual technology, and the method comprises the steps: obtaining a plurality of target operation areas in an industrial robot operation scene and the visual image data of a plurality of industrial robots; determining a target feature vector of each target operation area and a state feature vector of each industrial robot based on the visual image data; and according to the target feature vectors of the target operation areas, the movement directions of the target operation areas relative to the industrial robot, the surface deformation conditions of the target operation areas and the relative position change conditions of the target operation areas and the tail end operation component of the industrial robot are determined. According to the method, the state characteristics of the target operation area and the industrial robot are accurately determined by utilizing the visual image data, the interaction between the operation area and the robot can be effectively analyzed and predicted, the operation failure caused by errors or accidental interference can be reduced, and the working efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, specifically to a control method and apparatus for industrial robots based on vision technology. Background Technology

[0002] Currently, traditional methods typically rely on preset programs and fixed operating procedures. This makes robots slow to react when faced with complex environments or unforeseen changes, and they are prone to low work accuracy. This not only affects work efficiency but also increases the risk of failure due to errors or interference. Moreover, robots usually lack sufficient dynamic feedback mechanisms and cannot monitor their interaction with the target area in real time. Therefore, robots have difficulty predicting and identifying potential interference in a timely manner, which may lead to unforeseen interference or conflict during operation, and may even require shutdown for inspection.

[0003] Furthermore, when multiple robots need to work collaboratively, traditional methods often struggle to efficiently allocate tasks and priorities among them. This can easily lead to resource waste and operational conflicts, impacting production efficiency. Collaboration between multiple robots largely relies on manual scheduling, lacking intelligent task optimization and dynamic adjustment mechanisms. Moreover, robots typically operate according to fixed control procedures, lacking the ability to adapt to real-time environmental changes. In complex environments or specialized tasks, robots using traditional methods cannot effectively adjust, resulting in poor operational flexibility. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides the following technical solution: a control method for industrial robots based on vision technology, comprising: Acquire visual image data of multiple target work areas and multiple industrial robots in an industrial robot operation scenario; based on the visual image data, determine the target feature vector of each target work area and the state feature vector of each industrial robot. Based on the target feature vector of each target work area, determine the motion direction of each target work area relative to the industrial robot, the surface deformation of the target work area, and the relative position change between the target work area and the end effector of the industrial robot. Based on the state feature vectors of each industrial robot, the collaborative operation relationship between the industrial robots and the operation priority of each industrial robot are determined. Based on the motion direction of each target work area relative to the industrial robot, the surface deformation of the target work area, the relative position change, the cooperative work relationship and work priority between each industrial robot, predict whether each industrial robot will interfere with the target work area when performing the work. When interference is predicted, determine the expected time of occurrence of the interference and the industrial robots and target work areas involved in the interference; When the expected occurrence time is less than a preset time threshold, the adjustment amount of each joint rotation of each industrial robot involved in the interference is determined based on the interference situation, the industrial robots involved in the interference, and the target working area. The dynamic feedback control mechanism of each industrial robot involved in the interference is triggered based on the adjustment amount.

[0005] Preferably, the target feature vector includes color features, texture features, shape features, and spatial location features of the target working area, and the state feature vector includes the position features, posture features, and motion trend features of the industrial robot; Based on visual image data, the target feature vectors of each target work area and the state feature vectors of each industrial robot are determined, including: Determine the transformation mapping relationship between the image coordinate system and the global coordinate system of the industrial robot's operating scene; Based on the transformation mapping relationship, coordinate transformation is performed on the target feature points of each target work area and the state feature points of each industrial robot in the visual image data to obtain the target feature vector of each target work area and the state feature vector of each industrial robot.

[0006] Preferably, based on the target feature vector of each target work area, the motion direction of each target work area relative to the industrial robot, the surface deformation of the target work area, and the relative positional change between the target work area and the end effector of the industrial robot are determined, including: Determine the target feature vectors for each target work area within a preset time window; The target feature vectors of each target work area within a preset time window are comprehensively analyzed to obtain the displacement direction information of the target work area within the time window. The displacement direction information is used to characterize the motion direction of the target work area relative to the industrial robot.

[0007] Preferably, based on the target feature vectors of each target work area, the motion direction of each target work area relative to the industrial robot, the surface deformation of the target work area, and the relative positional change between the target work area and the end effector of the industrial robot are determined, further comprising: Determine the target feature vectors for each target work area within a preset time window; Based on the target feature vectors of each target work area within the preset time window, analyze the surface morphology changes of the target work area within the time window to obtain the surface deformation of each target work area. Determine the target feature vectors for each target work area within a preset time window; Based on the target feature vectors of each target work area within the preset time window, the positional changes of the target work area and the end effector of the industrial robot within the time window are calculated, thus obtaining the relative positional changes of each target work area and the end effector of the industrial robot.

[0008] Preferably, the collaborative operation relationship between industrial robots is determined based on the state feature vectors of each industrial robot, including: Based on the positional and orientation characteristics of each industrial robot, the spatial distribution of each industrial robot in the work scenario is analyzed; By combining the motion trend characteristics of each industrial robot, we can determine whether there is a correlation in the work tasks among the industrial robots and determine the collaborative work relationship among them.

[0009] Preferably, the task priority of each industrial robot is determined based on its state feature vector, including: Based on the positional characteristics of each industrial robot, determine the distance between each industrial robot and the key work point; Based on the motion trend characteristics and task types of each industrial robot, and according to the distance from the critical work point and the importance of the task, the task priority of each industrial robot is determined.

[0010] Preferably, based on the motion direction of each target work area relative to the industrial robot, the surface deformation of the target work area, the relative position change, and the cooperative working relationship and work priority among the industrial robots, it is predicted whether each industrial robot will interfere with the target work area when performing the work, including: The system determines whether the target work area moves towards the industrial robot work area, whether the surface deformation of the target work area exceeds the preset deformation range, and whether the relative position change between the target work area and the end effector of the industrial robot indicates that the distance between them is continuously decreasing. The duration of these conditions being met is longer than the preset judgment time. In addition, the system combines the collaborative operation relationship and operation priority between the industrial robots to determine whether interference will occur. If it is determined that the above conditions are met and there is an interference risk based on the collaborative operation relationship and operation priority analysis, it is predicted that each industrial robot will interfere with the target operation area when performing the operation.

[0011] Preferably, determining the expected time of occurrence of the interference includes: Determine the distance between the target work area and the origin of the industrial robot's body coordinate system, the distance between the target work area and the end effector of the industrial robot, and the motion velocity vector of the target work area; The predicted time of interference is obtained by comprehensively calculating the distance between the target work area and the origin of the industrial robot's body coordinate system, the distance between the target work area and the end effector of the industrial robot, and the motion velocity vector of the target work area.

[0012] Preferably, after triggering the dynamic feedback control mechanism of each industrial robot involved in the interference according to the adjustment amount, the method further includes: Activate the real-time visual monitoring module to continuously acquire visual feedback information during the job execution process; Extract the actual motion direction data of each target work area, the actual surface deformation data of each target work area, the actual relative position data of each target work area and the end effector of the industrial robot, and the actual rotation data of each joint of the industrial robot from the visual feedback information. The actual motion direction data is compared with the predicted motion direction of the target working area; the actual surface deformation data is compared with the predicted surface deformation based on the surface deformation of the target working area; the actual relative position data is compared with the predicted relative position based on the relative position change; and the actual rotation data of each joint of each industrial robot is compared with the determined rotation adjustment amount of each joint. If any data exceeds the corresponding comparison range, the parameters determined based on the target feature vector of the target working area and the state feature vector of the industrial robot are readjusted according to the deviation, and the rotation adjustment amount of each joint of the industrial robot involved in the interference is re-determined.

[0013] A control device for an industrial robot based on vision technology, applicable to the aforementioned control method for an industrial robot based on vision technology, includes: The feature extraction unit is used to acquire visual image data of multiple target work areas and multiple industrial robots in the industrial robot operation scenario; based on the visual image data, it determines the target feature vector of each target work area and the state feature vector of each industrial robot. The change determination unit is used to determine the motion direction of each target work area relative to the industrial robot, the surface deformation of the target work area, and the relative position change between the target work area and the end effector of the industrial robot, based on the target feature vector of each target work area. The state detection unit is used to determine the collaborative operation relationship between industrial robots and the operation priority of each industrial robot based on the state feature vector of each industrial robot. The interference judgment unit is used to predict whether each industrial robot will interfere with the target work area when performing the operation, based on the movement direction of each target work area relative to the industrial robot, the surface deformation of the target work area, the relative position change, the cooperative operation relationship and operation priority between each industrial robot. The interference prediction unit is used to determine the expected time of interference and the industrial robot and target work area involved in the interference when interference is predicted to occur. The rotation adjustment unit is used to determine the rotation adjustment amount of each joint of each industrial robot involved in the interference, based on the interference situation, the industrial robot involved in the interference, and the target working area, when the expected occurrence time is less than a preset time threshold, and to trigger the dynamic feedback control mechanism of each industrial robot involved in the interference based on the adjustment amount.

[0014] Compared with the prior art, the beneficial effects of the present invention are: (1) By utilizing visual image data, this invention can accurately determine the state characteristics of the target work area and the industrial robot, and effectively analyze and predict the interaction between the work area and the robot. This method can reduce work failures caused by errors or accidental interference and improve work efficiency. Furthermore, by analyzing the target feature vector of the work area, the interference between the robot and the target work area can be predicted, thereby discovering potential interference problems in advance and making real-time adjustments through a dynamic feedback control mechanism. This not only reduces the risks in the operation, but also enables the robot to react quickly according to environmental changes, thereby avoiding accidental interference. (2) By analyzing the collaborative operation relationship and operation priority between industrial robots, this invention can reasonably arrange the operation tasks and priorities when multiple robots work together, ensuring that each robot can cooperate efficiently, avoiding ineffective resource waste and potential conflicts, improving productivity, and during the operation, real-time monitoring of visual feedback information and comparison with predicted data can promptly detect whether various parameters have deviated and readjust the robot's actions. This closed-loop control system significantly improves the adaptability and operational flexibility of the robot, especially in complex environments, it can quickly adjust to meet new challenges. (3) By predicting possible interference situations in advance and adjusting according to the time and range of interference, the present invention can effectively avoid conflicts between the robot and the target work area during the execution of tasks, and reduce safety hazards caused by operational errors or unpredictable factors; and by monitoring the surface deformation and relative position changes of the target work area, the robot can adapt to environmental changes more accurately, ensuring smoother and safer interaction with complex work environments when performing tasks. Attached Figure Description

[0015] Figure 1 This is a schematic flowchart of the overall method in one embodiment of the present invention; Figure 2 This is a schematic diagram of the overall device architecture in one embodiment of the present invention.

[0016] In the diagram: 1. Feature extraction unit; 2. Change determination unit; 3. State detection unit; 4. Interference judgment unit; 5. Interference prediction unit; 6. Rotation adjustment unit. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1, please refer to Figure 1 This invention provides a technical solution: a control method for industrial robots based on vision technology, comprising: S1. Acquire visual image data of multiple target work areas and multiple industrial robots in the industrial robot operation scenario; Based on the visual image data, determine the target feature vector of each target work area and the state feature vector of each industrial robot. S2. Based on the target feature vector of each target work area, determine the motion direction of each target work area relative to the industrial robot, the surface deformation of the target work area, and the relative position change between the target work area and the end effector of the industrial robot. S3. Based on the state feature vectors of each industrial robot, determine the collaborative operation relationship between each industrial robot and the operation priority of each industrial robot; S4. Based on the movement direction of each target work area relative to the industrial robot, the surface deformation of the target work area, the relative position change, and the cooperative work relationship and work priority between each industrial robot, predict whether each industrial robot will interfere with the target work area when performing the work. S5. When interference is predicted, determine the expected time of occurrence of the interference and the industrial robots and target work areas involved in the interference; S6. When the expected occurrence time is less than the preset time threshold, determine the joint rotation adjustment amount of each joint of each industrial robot involved in the interference based on the interference situation, the industrial robots involved in the interference, and the target working area, and trigger the dynamic feedback control mechanism of each industrial robot involved in the interference based on the adjustment amount.

[0019] It should be noted that visual sensors (such as cameras, LiDAR, etc.) are used to collect image data of the work area and obtain the target features (e.g., the position and shape of the workpiece) and the state of each robot (e.g., the robot's position, posture, and actions). For example, suppose there are two robots in areas A and B of the production line. Area A has an assembly operation, and area B has a handling task. Robots A and B capture image data of the target areas through cameras and extract the target features (such as the geometry and surface features of the workpiece) of their respective work areas through image processing algorithms. At the same time, the state features of the robots (such as their current position and action status) are also captured. By utilizing the target feature vector of image data (i.e., features extracted from the image), the relationship between each work area and the robot can be determined. Specifically, this includes the direction of motion of the work area relative to the robot, surface deformation, and the relative positional change of the robot's end effector to the work area. For example, robot A is performing an assembly task, while robot B in area B is handling a workpiece. Through a vision sensor, the system can calculate the surface deformation of the workpiece in area A (such as bending or movement of the workpiece) and the direction of motion of robot B relative to area A, thus determining the relative position of robot B and the workpiece in area A. The system analyzes the collaborative relationships between robots, such as which robot is responsible for which task and which robot should complete its task first, to further determine the priority of each robot. For example, robot A is responsible for assembly work, and robot B is responsible for moving workpieces. By analyzing the nature and progress of their tasks, the system determines that robot A's task has a higher priority than robot B's task, and robot B must wait for robot A to complete its assembly task before it can proceed to the next step of moving the workpieces. Based on the above information, the system predicts whether there will be interference between the robot and the target work area. Interference may refer to collisions or mutual interference between the robot and the target area or other robots. For example, suppose robot A and robot B are performing tasks at the same time, and the end effector of robot A is about to approach the work area of ​​area B. The system calculates the motion trajectory, task priority, and task direction of both robots to predict whether robot A will interfere with the workpiece in area B. If interference is predicted, the system proceeds to the next step. When the system predicts interference, the next step is to calculate the specific time when the interference will occur, as well as the robots and work areas involved. For example, if the system detects that the end effector of robot A will collide with the workpiece in area B at T+3 seconds, the system records this time and identifies the robots A and the workpiece in area B involved in the interference. If the expected time of interference is within a predetermined time threshold (e.g., 5 seconds), the system will adjust the robot's movements according to the interference situation; for example, adjusting the robot's joint angles to avoid interference. For instance, when the system determines that robot A will collide with the workpiece in area B within 3 seconds, the system will calculate the joint rotation adjustment of robot A and precisely adjust the movement trajectory of robot A to avoid interference. The dynamic feedback control mechanism will automatically adjust the movement of each joint of robot A to ensure that robot A can safely avoid the interference area.

[0020] In an optional embodiment, the target feature vector includes color features, texture features, shape features, and spatial location features of the target working area, and the state feature vector includes the position features, posture features, and motion trend features of the industrial robot; Based on visual image data, the target feature vectors of each target work area and the state feature vectors of each industrial robot are determined, including: Determine the transformation mapping relationship between the image coordinate system and the global coordinate system of the industrial robot's operating scene; Based on the transformation mapping relationship, coordinate transformation is performed on the target feature points of each target work area and the state feature points of each industrial robot in the visual image data to obtain the target feature vector of each target work area and the state feature vector of each industrial robot.

[0021] It should be noted that the target feature vector is used to describe various features within the work area, typically including: color features: the characteristics of color distribution within the area, reflecting the physical properties or markings of the work area; texture features: the texture pattern of the surface within the area, such as whether it is smooth, whether it has lines, whether it has defects, etc.; shape features: the geometric shape of the area, whether it is regular, whether it has edges or curved surfaces, etc.; spatial position features: the position coordinates of the area in three-dimensional space, which helps to determine the spatial distribution and depth information of the target area; the state features of industrial robots typically include: position features: the current position coordinates (x, y, z) of the robot in the work scene; posture features: the direction and angle (pitch angle, yaw angle, etc.) of the robot's end effector (such as the end of a robotic arm) relative to the ground; motion trend features: the robot's motion direction and speed, usually obtained by analyzing the state of the previous few frames (e.g., whether the robot is accelerating, decelerating, or moving at a constant speed). Image coordinate systems are typically two-dimensional coordinate systems defined based on the camera's viewpoint, while industrial robot operating scenarios use a three-dimensional global coordinate system. Therefore, it is necessary to define the transformation mapping relationship between the image coordinate system and the global coordinate system. This transformation relationship is usually obtained through calibration methods, which match the coordinate systems by determining parameters such as the camera's position, viewpoint, and lens distortion. For example, suppose there is a camera mounted above a machine in the production line environment. The camera's image coordinate system is two-dimensional, for example, an image with a width of 800 pixels and a height of 600 pixels, while the global coordinate system is three-dimensional, where the x-axis represents the horizontal direction, the y-axis represents the vertical direction, and the z-axis represents the depth direction. In order to interface the image data captured by the camera with the actual operating scenario of the industrial robot, it is necessary to define the transformation formula between the image coordinate system and the global coordinate system using the camera's internal and external parameters. Once the transformation relationship between the image coordinate system and the global coordinate system is determined, coordinate transformations can be performed on target feature points and robot state feature points in the image data. Target feature point transformation involves extracting key points (such as workpiece edges, specific marker points, etc.) from the image, and then using coordinate transformation formulas to transform these points from the image coordinate system to the global coordinate system. State feature point transformation similarly requires transforming the robot's position and orientation from the image coordinate system to the global coordinate system. For example, the position of robot A is determined by a vision sensor, which may be located in the robot's relative coordinate system. The system acquires image data and transforms it to convert the location into its true location in the global coordinate system. For example, suppose a feature point in the target work area (such as a corner of a workpiece) in the image corresponds to a point (400, 300) in the image coordinate system, while the position of industrial robot A in the camera's view is (150, 120). Through the transformation mapping relationship after camera calibration, the global coordinates (X, Y, Z) of that point can be obtained. For example, the global position of the workpiece corner is (2.5, 3.0, 0.8), while the global position of robot A is (1.0, 0.5, 0.3). By transforming the target feature points and state feature points, the system can construct the target feature vector of the target working area and the state feature vector of the industrial robot. The target feature vector includes information such as color, texture, shape, and spatial position, and is usually a multi-dimensional vector containing all the important features of the target working area. The state feature vector includes information such as the robot's position, attitude, and motion trend, and is also a multi-dimensional vector that reflects the robot's motion state. For example, suppose the target working area contains three main features: the color feature vector is [0.8, 0.2, 0.5] (representing RGB color information), the texture feature vector is [0.1, 0.3, 0.5] (representing texture coarseness), and the shape feature vector is [0.4, 0.7] (representing the shape features of the area). The robot's state feature vector may include: position [1.0, 0.5, 0.3] (global coordinates), attitude [45°, 10°] (pitch and yaw angles), and motion trend [0.2, 0.4] (representing the robot's current speed and direction).

[0022] In an optional embodiment, based on the target feature vector of each target work area, the motion direction of each target work area relative to the industrial robot, the surface deformation of the target work area, and the relative positional change between the target work area and the end effector of the industrial robot are determined, including: Determine the target feature vectors for each target work area within a preset time window; The target feature vectors of each target work area within a preset time window are comprehensively analyzed to obtain the displacement direction information of the target work area within the time window. The displacement direction information is used to characterize the motion direction of the target work area relative to the industrial robot.

[0023] It's important to note that the target feature vector is a vector describing the color, texture, shape, and spatial location of the target work area (such as a workpiece). To analyze information such as the movement direction of the target work area, the target feature vector needs to be collected and processed within a preset time window. A preset time window refers to selecting a fixed time period (e.g., 1 second, 2 seconds, 5 seconds, etc.) within which the target features of the work area are collected and analyzed. This data will be used for subsequent analysis of the target's movement trends. Example: Suppose we are monitoring an industrial robot that is moving workpieces on a production line. The set time window is 1 second, which means that the target work area's features will be collected every second. The feature data of the domain (e.g., workpiece position, shape, color, etc.); assuming that in the first second, the feature vector of the target area might be: color feature: [0.5, 0.6, 0.7] (RGB values), shape feature: [0.4, 0.8] (indicating the shape characteristics of the area), spatial location feature: [1.2, 2.5, 0.3] (indicating the position of the target area in three-dimensional space), then, in the second second, the feature vector of the area is collected again, and it might become: color feature: [0.6, 0.7, 0.8], shape feature: [0.5, 0.7], spatial location feature: [1.3, 2.6, 0.4]. These feature vectors will represent the state of the target area within the time window; By comprehensively analyzing the feature vectors of the target work area within a time window, the displacement direction information of the target area can be obtained, that is, the direction of movement of the target area relative to the industrial robot during this time period. The displacement direction information is obtained by comparing the spatial position of the target feature vectors at different times (such as the position coordinates of the target area). If the target position coordinates change, the direction and speed of the target's movement can be inferred. For example, in the first time window (1 second), the spatial position of the target area is [1.2, 2.5, 0.3]; in the second time window (1 second), the spatial position of the target area is [1.3, 2.6, 0.4]. By calculating the difference between these two positions, the displacement of the target is obtained: displacement vector = [1.3-1.2, 2.6-2.5, 0.4-0.3] = [0.1, 0.1, 0.1]. Based on the displacement direction information obtained above, we can further analyze the relative positional changes between the target work area and the industrial robot, as well as the surface deformation of the target area (e.g., whether the workpiece has been compressed, stretched, or bent). Direction of motion: The displacement direction information reveals the direction of motion of the target work area relative to the robot's end effector. Surface deformation: Surface deformation can be inferred from changes in texture and shape features. For example, significant changes in texture features may indicate surface deformation of the target area. Relative positional changes: If the posture and position of the industrial robot's end effector change, the relative position between the target work area and the robot will also change accordingly. This is usually expressed through changes in spatial coordinates. Example: Assume the spatial position of the industrial robot's end effector is [1.0, 2.0, 0.5], while the spatial position of the target work area at a certain moment is [1.2, 2.5, 0.3]. If the robot's end effector does not move, but the target work area does, the direction of motion of the target relative to the robot can be inferred from the change in the target's position. By comparing the displacement direction of the target area with the relative position of the robot, the motion trend of the target can be obtained, such as whether the target area is approaching or moving away from the robot's end effector.

[0024] In an optional embodiment, based on the target feature vectors of each target work area, the motion direction of each target work area relative to the industrial robot, the surface deformation of the target work area, and the relative positional change between the target work area and the end effector of the industrial robot are determined, and the method further includes: Determine the target feature vectors for each target work area within a preset time window; Based on the target feature vectors of each target work area within the preset time window, analyze the surface morphology changes of the target work area within the time window to obtain the surface deformation of each target work area. Determine the target feature vectors for each target work area within a preset time window; Based on the target feature vectors of each target work area within the preset time window, the positional changes of the target work area and the end effector of the industrial robot within the time window are calculated, thus obtaining the relative positional changes of each target work area and the end effector of the industrial robot.

[0025] It should be noted that the target feature vector is used to describe the key features of the target working area, such as location, shape, color, and texture. These target feature vectors need to be collected and stored within a preset time window; this data will form the basis for subsequent analysis. For example, assuming the set time window is 1 second, the target area is a workpiece, and an industrial robot performs a grasping task in this area; every second, the system collects the target feature vector of the target area: In the first second, the feature vector of the target area might be: Spatial location: [1.2, 2.5, 0.3] (representing the location of the target area), Shape feature: [0.4, 0.8] (e.g., the aspect ratio of the workpiece), Color feature: [0.5, 0.6, 0.7] (RGB color values). In the second second, the feature vector of the target area might become: Spatial location: [1.3, 2.6, 0.4], Shape feature: [0.5, 0.9], Color feature: [0.6, 0.7, 0.8]. These feature vectors will be updated within each time window, reflecting the changes in the target area. By analyzing the surface morphology-related data (such as shape features and texture features) in the target feature vector, the surface deformation of the target working area within a time window can be obtained. If the target surface changes (such as compression, stretching, or bending of the workpiece), these changes will be reflected in the shape and texture parts of the feature vector. For example, suppose a workpiece is being monitored, and its shape and texture change within a time window: In the first second, the workpiece's shape feature is [0.4, 0.8], indicating that the workpiece's aspect ratio is 0.4:0.8; in the second second, the workpiece's shape feature is [0.5, 0.9], indicating that the workpiece's shape has become slightly longer and wider. By comparing the shape feature vectors at these two time points, it can be concluded that the surface of the target working area has undergone some degree of deformation; for example, the workpiece's surface may have been stretched due to pressure or external factors. Calculate the relative positional changes of the target work area and the end effector (e.g., robotic arm, gripper) of the industrial robot within a time window. Typically, the position of the end effector also changes, making its relative position to the target work area crucial. For example, suppose the initial position of the end effector is [1.0, 2.0, 0.5], and the initial position of the target work area is [1.2, 2.5, 0.3]. Within the first time window (1 second), the position of the target area changes to [1.3, 2.6, 0.4]. Simultaneously, the position of the end effector changes to [...]. [1.1, 2.1, 0.55]; By calculating the relative position between the target work area and the robot's end effector, the following results can be obtained: Initially, the relative position is: relative position = [1.2-1.0, 2.5-2.0, 0.3-0.5] = [0.2, 0.5, -0.2]. After the second time window, the relative position is: relative position = [1.3-1.1, 2.6-2.1, 0.4-0.55] = [0.2, 0.5, -0.15]. By comparing the changes in these relative positions, the relative movement trend between the target work area and the industrial robot's end effector can be obtained.

[0026] In an optional embodiment, determining the collaborative operation relationship between the industrial robots based on the state feature vectors of each industrial robot includes: Based on the positional and orientation characteristics of each industrial robot, the spatial distribution of each industrial robot in the work scenario is analyzed; By combining the motion trend characteristics of each industrial robot, we can determine whether there is a correlation in the work tasks among the industrial robots and determine the collaborative work relationship among them.

[0027] It should be noted that the spatial distribution of industrial robots can be determined through their positional and orientation features. Positional features typically refer to the coordinates of the robot's end effector, while orientation features describe the direction or posture of the end effector. Analyzing these features helps determine the specific positions of each robot in the working environment and the spatial relationships between them. For example, suppose there are three industrial robots performing different assembly tasks on an automated assembly line: Robot A: Positional features [3.0, 4.0, 1.5], Orientation features [0, 90, 0] (indicating that Robot A's end effector rotates 90 degrees along the Z-axis); Robot B: Positional features [7.0, 4.0, 1.5], Orientation features [0, 0, 0] (indicating that Robot B's end effector rotates 90 degrees along the Z-axis); (The actuator is not rotating); Robot C: Position characteristics are [10.0, 2.0, 1.5], and attitude characteristics are [0, 180, 0] (indicating that the end effector of Robot C rotates 180 degrees along the Z-axis); By analyzing these position and attitude characteristics, the following spatial distribution can be obtained: Robot A and Robot B are relatively close in position and may be performing tasks in the same area. Their attitude characteristics also show that their tasks may be different (Robot A is rotating); Robot C is far away from Robots A and B and may be performing tasks in different areas. Its attitude characteristics show that it may be performing tasks that require large-angle rotation; Based on these analyses, the spatial distribution of each robot in the work scenario can be determined so as to subsequently determine whether they have the possibility of collaborative operation; By analyzing the motion trend characteristics of each industrial robot, we can understand their motion patterns, such as movement speed and direction. Combining this information, we can determine whether the robots are working on the same task or whether they need to collaborate to complete a task. For example, let's continue with three robots: Robot A: Position characteristics [3.0, 4.0, 1.5], Motion trend characteristics [0.1, 0, 0] (indicating that Robot A moves slowly along the X-axis); Robot B: Position characteristics [7.0, 4.0, 1.5], Motion trend characteristics [-0.1, 0, 0] (indicating that Robot B moves slowly in the opposite direction along the X-axis); Robot C: Position characteristics [10.0, 2.0, 1.5], Motion trend characteristics [0, 0.2, 0] (indicating that Robot C moves along the Y-axis). From these motion trend characteristics, we can infer the following: Robots A and B: They move along the X-axis with opposite velocities. Given their proximity and opposite motion directions, this suggests they may need to cooperate, such as transferring items or working together to complete a specific task (e.g., performing an assembly operation simultaneously). Robot C: It moves along the Y-axis and appears to have no direct connection to the tasks of A and B. Based on its motion trend, it may perform a separate task, such as handling another task in a different area. Based on this analysis, we can conclude that: Robots A and B have a collaborative working relationship; they may need to complete an assembly task together or achieve a task goal through positional and motion coordination. Robot C does not have a direct collaborative working relationship with other robots and may perform a different task than A and B. After determining the motion trends between the industrial robots, it is also necessary to determine how they will cooperate. Typically, cooperative operations include synchronous operation, coordinated operation, and mutual backup. Specific examples include: Cooperative operation of robots A and B: Suppose A and B need to move a workpiece from one area of ​​an assembly line to another; A is responsible for gripping and moving the workpiece at the front end, while B is responsible for receiving it at the rear end for further processing. Since the motion trends of A and B are opposite, their movements need to be precisely coordinated to ensure that the workpiece is not dropped or damaged. In this case, the cooperative operation could be: Synchronous action: A and B need to coordinate the timing and speed of their movements; Coordinated operation: After A completes gripping, B begins to prepare to receive the workpiece. Independent operation of robot C: Robot C may independently perform another task, such as processing at the other end of an assembly line; its motion trend is along the Y-axis, meaning it will not interfere with or need to coordinate with A or B; C's task does not require cooperation with other robots.

[0028] In an optional embodiment, the task priority of each industrial robot is determined based on its state feature vector, including: Based on the positional characteristics of each industrial robot, determine the distance between each industrial robot and the key work point; Based on the motion trend characteristics and task types of each industrial robot, and according to the distance from the critical work point and the importance of the task, the task priority of each industrial robot is determined.

[0029] It should be noted that the distance between an industrial robot and a critical work point is an important factor in determining the priority of a task; a critical work point can be a specific location on the production line, a workstation, or a task that needs to be handled immediately; the closer the robot is to the critical work point, the higher its task priority is usually. Besides location, robot movement trends and task type also affect job priority. Movement trend characteristics typically include the robot's direction and speed of movement; while the task type determines its importance. For example, if a task is urgent or crucial to the overall production process, the robot performing that task should be given higher priority. Specific examples: Robot A: Current task is precision assembly, a high-priority task; movement trend characteristics [0.1,0,0], indicating that Robot A is moving slowly forward, and the task is progressing slowly. Robot B: Current task is material handling, a medium-priority task; movement trend characteristics [0.2,0,0], indicating that Robot B is moving forward at a relatively fast speed. Robot C: Current task is inspection, a low-priority task; movement trend characteristics [0.05,0,0], indicating that Robot C is moving very slowly, and the task's urgency is low. In this case, it is possible to... The task priorities are determined as follows: Robot A's task is precision assembly, a high-priority task. However, its movement speed is slow, and it is far from the critical work point (approximately 5.1 units), thus its task priority is somewhat affected, but it is still higher than other tasks. Robot B's task is material handling. Although it is a medium-priority task, its proximity to the critical work point and its fast movement speed (0.2 units / second) result in a very high task priority, especially when the task needs to be completed quickly. Robot C's task is inspection, a low-priority task with a very slow movement speed and a relatively far distance from the critical work point, thus its task priority is low. Therefore, considering both distance and task importance, the following priority ranking can be derived: Robot B (closest and most urgent); Robot A (high task priority, but far away and slow movement); Robot C (low task priority, slow movement, and far away). Besides task type and distance, other factors such as task urgency, robot health status, and workload can also affect priority determination; dynamic adjustments are needed based on these factors. For example, suppose robot A experiences a minor malfunction during operation, causing its execution speed to decrease; if the repair time is long, its task may be postponed; in this case, although robot A's task priority is high, robot B's priority may be temporarily increased due to its malfunction to ensure that the production line is not affected.

[0030] In an optional embodiment, based on the movement direction of each target work area relative to the industrial robot, the surface deformation of the target work area, the relative position change, and the cooperative working relationship and work priority among the industrial robots, it is predicted whether each industrial robot will interfere with the target work area when performing the work, including: The system determines whether the target work area moves towards the industrial robot work area, whether the surface deformation of the target work area exceeds the preset deformation range, and whether the relative position change between the target work area and the end effector of the industrial robot indicates that the distance between them is continuously decreasing. The duration of these conditions being met is longer than the preset judgment time. In addition, the system combines the collaborative operation relationship and operation priority between the industrial robots to determine whether interference will occur. If it is determined that the above conditions are met and there is an interference risk based on the collaborative operation relationship and operation priority analysis, it is predicted that each industrial robot will interfere with the target operation area when performing the operation.

[0031] It's important to note that determining whether the target work area faces the industrial robot's work area is crucial, especially when the target area's movement direction is directly towards the robot, as this increases the risk of interference. Additionally, surface deformation of the target work area must be considered. For example, changes in the target area's physical shape during robot operation could lead to collisions. A specific example: Suppose there are two robots in a workshop: Robot A and Robot B; the target work area is a flexible workbench performing laser cutting; Robot A is approaching the workbench for precision assembly, while Robot B is performing a transport task; the target work area (workbench) is moving towards Robot A's work area; assuming the workbench moves from left to right, and Robot A is currently to the left of the workbench; the relative position changes: the relative position between the workbench and Robot A is constantly changing, and as the workbench moves to the right, the distance between them gradually decreases; in this situation, if the workbench continues to approach Robot A, it may trigger an interference risk. The target work area may deform during task execution, especially in tasks involving flexible materials or complex assembly. Deformation exceeding a preset range may lead to collisions with the robot's end effector (such as a robotic arm). For example, suppose robot A's end effector is performing precision assembly, and the target work area is a plastic sheet that the robot needs to precisely place in a fixed position. During this process, the plastic sheet may undergo slight deformation (e.g., bending or expansion) due to laser cutting or temperature changes. If the deformation exceeds a preset range, robot A's actuator may interfere with the deformed plastic sheet. If the distance between the target work area and the robot continues to decrease for a period longer than the preset judgment time, it indicates that the proximity between the two has reached a high critical value, and the risk of interference is relatively high. For example, suppose robot B is performing a material handling task, and the target work area is an assembly table; robot B's path intersects the movement direction of the assembly table, and the robot's movement speed is relatively fast, and the target area has a large shape; if the relative position between robot B and the assembly table continues to decrease over the next 5 seconds, and this period exceeds the preset danger time (e.g., 2 seconds), then the risk of interference will become high. When multiple robots work collaboratively, if one robot has a higher task priority, the task execution order of other robots may need to be adjusted to avoid interference. For example, when tasks conflict, it is necessary to determine which robot should perform its task first based on the collaborative relationship between the robots. Specifically, suppose robot A's task is to assemble high-precision parts, which has a high priority; robot B's task is to transport materials, which has a medium priority. If the working areas of robots A and B overlap at the same time, and their movement directions are close, interference may occur. In this case, based on the collaborative working relationship, robot B's task priority is lower, and the system can adjust robot B's path or delay its task to ensure that robot A can complete its high-priority assembly task on time, avoiding interference. By analyzing the above aspects, a comprehensive prediction can be derived. For example, based on the distance between the robot and the target work area, the movement trend, the deformation of the target area, and the collaborative operation relationship, the system predicts whether there is a risk of interference. Example: Robot A: Performing precision assembly, high task priority; the target work area is moving towards Robot A, and the distance is gradually decreasing; the deformation of the target area may interfere with Robot A's operation for more than 2 seconds; considering task priority, Robot A has a higher priority, and the system needs to avoid interference as much as possible. Robot B: Performing material handling, medium task priority; the relative position change between Robot B and the target work area is small, and its path is relatively separated from the movement direction of the target area; therefore, Robot B has a lower task priority and will not immediately affect the interference prediction. Ultimately, the system determines that interference may occur in certain situations (such as excessive workbench deformation or when Robot A is too close to the target work area); therefore, the system will take corresponding measures, such as adjusting the robot's path or pausing certain tasks, to avoid interference.

[0032] In an optional embodiment, determining the expected time of occurrence of the interference includes: Determine the distance between the target work area and the origin of the industrial robot's body coordinate system, the distance between the target work area and the end effector of the industrial robot, and the motion velocity vector of the target work area; The predicted time of interference is obtained by comprehensively calculating the distance between the target work area and the origin of the industrial robot's body coordinate system, the distance between the target work area and the end effector of the industrial robot, and the motion velocity vector of the target work area.

[0033] In an optional embodiment, after triggering the dynamic feedback control mechanism of each industrial robot involved in the interference based on the adjustment amount, the method further includes: Activate the real-time visual monitoring module to continuously acquire visual feedback information during the job execution process; Extract the actual motion direction data of each target work area, the actual surface deformation data of each target work area, the actual relative position data of each target work area and the end effector of the industrial robot, and the actual rotation data of each joint of the industrial robot from the visual feedback information. The actual motion direction data is compared with the predicted motion direction of the target working area; the actual surface deformation data is compared with the predicted surface deformation based on the surface deformation of the target working area; the actual relative position data is compared with the predicted relative position based on the relative position change; and the actual rotation data of each joint of each industrial robot is compared with the determined rotation adjustment amount of each joint. If any data exceeds the corresponding comparison range, the parameters determined based on the target feature vector of the target working area and the state feature vector of the industrial robot are readjusted according to the deviation, and the rotation adjustment amount of each joint of the industrial robot involved in the interference is re-determined.

[0034] Example 2, please refer to Figure 2 This invention provides a technical solution: a control device for an industrial robot based on vision technology, applicable to the aforementioned control method for an industrial robot based on vision technology, comprising: Feature extraction unit 1 is used to acquire visual image data of multiple target work areas and multiple industrial robots in the industrial robot operation scenario; based on the visual image data, it determines the target feature vector of each target work area and the state feature vector of each industrial robot. The change determination unit 2 is used to determine the motion direction of each target work area relative to the industrial robot, the surface deformation of the target work area, and the relative position change between the target work area and the end effector of the industrial robot, based on the target feature vector of each target work area. The state detection unit 3 is used to determine the collaborative operation relationship between industrial robots and the operation priority of each industrial robot based on the state feature vector of each industrial robot. Interference judgment unit 4 is used to predict whether each industrial robot will interfere with the target work area when performing the operation, based on the movement direction of each target work area relative to the industrial robot, the surface deformation of the target work area, the relative position change, the cooperative operation relationship and operation priority between each industrial robot. Interference prediction unit 5 is used to determine the expected time of interference and the industrial robot and target work area involved in the interference when interference is predicted to occur. The rotation adjustment unit 6 is used to determine the rotation adjustment amount of each joint of each industrial robot involved in the interference, based on the interference situation, the industrial robot involved in the interference, and the target working area, when the expected occurrence time is less than a preset time threshold, and to trigger the dynamic feedback control mechanism of each industrial robot involved in the interference based on the adjustment amount.

[0035] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A control method of an industrial robot based on vision technology, characterized in that, The method comprises: acquiring visual image data of a plurality of target work areas and a plurality of industrial robots in an industrial robot work scene; determining target feature vectors of the target work areas and state feature vectors of the industrial robots based on the visual image data; determining, according to the target feature vectors of the target work areas, motion directions of the target work areas relative to the industrial robots, surface deformation conditions of the target work areas, and relative position change conditions of the target work areas and end operating components of the industrial robots; determining, according to the state feature vectors of the industrial robots, cooperative work relationships between the industrial robots and work priorities of the industrial robots; predicting, according to the motion directions of the target work areas relative to the industrial robots, the surface deformation conditions of the target work areas, the relative position change conditions, the cooperative work relationships between the industrial robots, and the work priorities, whether an interference condition will occur when the industrial robots perform work; when the predicted occurrence time is less than a preset time threshold, determining, according to the interference condition, the industrial robots and the target work areas involved in the interference, joint rotation adjustment amounts of the industrial robots involved in the interference, and triggering a dynamic feedback control mechanism of the industrial robots involved in the interference according to the adjustment amounts. The target feature vectors comprise color features, texture features, shape features, and spatial position features of the target work areas, and the state feature vectors comprise position features, posture features, and motion trend features of the industrial robots; 2. The control method of an industrial robot based on vision technology according to claim 1, characterized in that, determining, based on the visual image data, the target feature vectors of the target work areas and the state feature vectors of the industrial robots comprises: determining a conversion mapping relationship between an image coordinate system and a global coordinate system of the industrial robot work scene; performing coordinate transformation on target feature points of each target work area and state feature points of each industrial robot in the visual image data according to the conversion mapping relationship to obtain the target feature vectors of the target work areas and the state feature vectors of the industrial robots. determining, according to the target feature vectors of the target work areas, the motion directions of the target work areas relative to the industrial robots, the surface deformation conditions of the target work areas, and the relative position change conditions of the target work areas and end operating components of the industrial robots comprises:

3. The control method of an industrial robot based on vision technology according to claim 2, characterized in that, determining target feature vectors of each target work area corresponding to a preset time window length; comprehensively analyzing the target feature vectors of each target work area corresponding to the preset time window length to obtain displacement trend information of the target work areas in the time window, wherein the displacement trend information is used to represent the motion directions of the target work areas relative to the industrial robots. determining, according to the target feature vectors of the target work areas, the motion directions of the target work areas relative to the industrial robots, the surface deformation conditions of the target work areas, and the relative position change conditions of the target work areas and end operating components of the industrial robots further comprises:

4. The control method of an industrial robot based on vision technology according to claim 3, characterized in that, ​ Determine the target feature vector of each target work area corresponding to the preset time window length; According to the target feature vector of each target work area corresponding to the preset time window length, analyze the surface shape change condition of the target work area in the time window, and obtain the surface deformation condition of each target work area; Determine the target feature vector of each target work area corresponding to the preset time window length; According to the target feature vector of each target work area corresponding to the preset time window length, calculate the position change of the target work area and the end operation component of the industrial robot in the time window, and obtain the relative position change of each target work area and the end operation component of the industrial robot.

5. The control method of an industrial robot based on vision technology according to claim 4, characterized in that, According to the state feature vector of each industrial robot, determine the cooperative work relationship between each industrial robot, including: According to the position feature and attitude feature of each industrial robot, analyze the spatial distribution of each industrial robot in the work scene; Combined with the motion trend feature of each industrial robot, judge whether there is a correlation between the work tasks of each industrial robot, and determine the cooperative work relationship between each industrial robot.

6. The control method of an industrial robot based on vision technology according to claim 5, characterized in that, According to the state feature vector of each industrial robot, determine the work priority of each industrial robot, including: According to the position feature of each industrial robot, determine the distance between each industrial robot and the key work point; According to the motion trend feature and work task type of each industrial robot, according to the distance from the key work point and the importance of the work task, determine the work priority of each industrial robot.

7. The control method of an industrial robot based on vision technology according to claim 6, characterized in that, According to the motion direction of each target work area relative to the industrial robot, the surface deformation condition of the target work area, the relative position change, and the cooperative work relationship and work priority between each industrial robot, predict whether there will be interference between each industrial robot and the target work area when performing work, including: Judge whether there is a target work area motion direction towards the industrial robot work area, a target work area surface deformation condition exceeding a preset deformation range, a target work area and an industrial robot end operation component relative position change indicating that the distance between them continues to decrease, and at the same time meet the above conditions for a duration greater than a preset judgment duration, and at the same time, combined with the cooperative work relationship and work priority between each industrial robot, judge whether there will be interference; If it is judged that the above conditions are met and there is an interference risk according to the cooperative work relationship and work priority, it is predicted that there will be interference between each industrial robot and the target work area when performing work.

8. The control method of an industrial robot based on visual technology according to claim 7, characterized in that, Determine the predicted occurrence time of the interference condition, including: Determine the distance between the target work area and the origin of the industrial robot body coordinate system, the distance between the target work area and the industrial robot end operation component, and the motion velocity vector of the target work area, respectively; According to the distance between the target work area and the origin of the industrial robot body coordinate system, the distance between the target work area and the industrial robot end operation component, and the motion velocity vector of the target work area, comprehensive operation is carried out to obtain the predicted occurrence time of the interference condition.

9. The control method of an industrial robot based on visual technology according to claim 8, characterized in that, After triggering the dynamic feedback control mechanism of the industrial robots involved in the interference according to the adjustment amount, the method further comprises: starting a real-time visual monitoring module to continuously acquire visual feedback information during the execution of the task; extracting actual motion direction data of each target work area, actual surface deformation data of each target work area, actual relative position data of each target work area and the end operating component of the industrial robot, and actual rotation data of each joint of each industrial robot from the visual feedback information; comparing the actual motion direction data with the predicted target work area motion direction, comparing the actual surface deformation data with the predicted surface deformation based on the surface deformation of the target work area, comparing the actual relative position data with the predicted relative position based on the relative position change, and comparing the actual rotation data of each joint of each industrial robot with the determined adjustment amount of each joint rotation; if there is data beyond the corresponding comparison range, re-adjusting each parameter determined based on the target feature vector of the target work area and the state feature vector of the industrial robot according to the deviation, and re-determining the adjustment amount of each joint rotation of each industrial robot involved in the interference.

10. A control device for a vision technology-based industrial robot, which is adapted to the control method for a vision technology-based industrial robot according to any one of claims 1 to 9, characterized in that, Comprise: a feature extraction unit for acquiring visual image data of a plurality of target work areas and a plurality of industrial robots in an industrial robot work scene; determining a target feature vector of each target work area and a state feature vector of each industrial robot based on the visual image data; a change determination unit for determining the motion direction of each target work area relative to the industrial robot, the surface deformation of the target work area, and the relative position change of the target work area and the end operating component of the industrial robot based on the target feature vector of each target work area, respectively; a state detection unit for determining the cooperative work relationship between the industrial robots and the work priority of each industrial robot based on the state feature vector of each industrial robot; an interference judgment unit for predicting whether an interference will occur between each industrial robot and the target work area when performing the work based on the motion direction of each target work area relative to the industrial robot, the surface deformation of the target work area, the relative position change, and the cooperative work relationship and work priority between the industrial robots; an interference prediction unit for determining the predicted occurrence time of the interference, the industrial robot and the target work area involved in the interference when predicting that the interference will occur; a rotation adjustment unit for determining the adjustment amount of each joint rotation of each industrial robot involved in the interference according to the interference, the industrial robot and the target work area involved in the interference when the predicted occurrence time is less than a preset time threshold, and triggering the dynamic feedback control mechanism of each industrial robot involved in the interference according to the adjustment amount.

Images