Method and system for controlling mechanical arm and storage medium

Abstract

The method for controlling the mechanical arm comprises the steps that images, collected by multiple cameras at the current moment, of multiple mechanical arm operation areas are processed to obtain the two-dimensional space positions of multiple human body detection areas and human body detection points; dividing the plurality of human body detection areas into one or more human body detection area sets based on a plurality of pieces of feature information extracted from the plurality of human body detection areas; the three-dimensional space position of the human body detection point at the current moment is determined based on the two-dimensional space position of the human body detection point in the human body detection area in each human body detection area set, and the three-dimensional space position of the human body detection point at the historical moment is selectively determined based on the three-dimensional space position of the human body detection point at the historical moment; determining a three-dimensional space position of each human body key point in the human body skeleton topological structure at the current moment; and the distance between each human body key point and the mechanical arm key point is determined based on the three-dimensional space position of each human body key point and the three-dimensional space position of the mechanical arm key point, and operation of the mechanical arm is controlled based on the distance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of control, and more particularly to a method of controlling a robot arm, a system of controlling a robot arm and a computer storage medium. BACKGROUND

[0002] With the increasing development of industrial automation, mixed production scenarios of robots cooperating with humans are becoming more and more common, such as scenarios of automobile production, electronic manufacturing, etc. In the mixed production scenarios of robots cooperating with humans, multiple workers and robot arms work in high frequency in a limited space, which puts high requirements on the safety of human-robot collaboration.

[0003] At present, the distance between multiple workers and a robot arm is monitored in real time by visual perception to ensure that the robot arm avoids collision with the workers while operating. However, the current visual perception monitoring scheme is difficult to apply to complex scenarios with a large number of personnel and occluded visual angles. For example, the current visual perception monitoring scheme is difficult to accurately identify and match multiple workers from different camera angles, and local occlusion of the robot arm or personnel, rapid movement of personnel may affect the accurate detection of the three-dimensional position of the key parts of the human body, and thus it is difficult to achieve safe control of the robot arm. SUMMARY

[0004] In order to solve or at least alleviate one or more of the above problems, the following technical solutions are provided.

[0005] According to a first aspect of the present application, a method of controlling a robot arm is provided, the method comprising the following steps: processing images of multiple robot operation areas collected by multiple cameras at a current time to obtain multiple human detection areas and two-dimensional spatial positions of human detection points in each human detection area; extracting multiple feature information from the multiple human detection areas and dividing the multiple human detection areas into one or more human detection area sets based on the extracted multiple feature information, wherein each human detection area set belongs to the same human object and includes multiple human detection areas; determining three-dimensional spatial positions of human detection points at the current time based on the three-dimensional spatial positions of the human detection points in the multiple human detection areas in each human detection area set, and optionally based on three-dimensional spatial positions of human detection points at historical times, determining three-dimensional spatial positions of each human key point in a human skeleton topology at the current time; and determining distances between each human key point and a robot key point based on the three-dimensional spatial positions of each human key point in the human skeleton topology at the current time and the three-dimensional spatial position of the robot key point, and controlling the operation of the robot arm based on the distances.

[0006] According to the method for controlling the mechanical arm, the method comprises: inputting the images of the plurality of mechanical arm operation regions captured by the plurality of cameras at the current time into a pose estimation network model to obtain the plurality of human detection regions and the two-dimensional spatial positions of the human detection points in each human detection region.

[0007] According to the method for controlling the mechanical arm, the method comprises: inputting the images of the plurality of mechanical arm operation regions captured by the plurality of cameras at the current time into a pose estimation network model to obtain the plurality of human detection regions and the two-dimensional spatial positions of the human detection points in each human detection region.

[0008] According to the method for controlling the mechanical arm, the method comprises: inputting the images of the plurality of mechanical arm operation regions captured by the plurality of cameras at the current time into a pose estimation network model to obtain the plurality of human detection regions and the two-dimensional spatial positions of the human detection points in each human detection region.

[0009] In an embodiment of the method for controlling the robot arm according to the present application or any of the above embodiments, wherein the determining the three-dimensional spatial position of each human key point in the current time's human skeleton topology based on the three-dimensional spatial position of the human detection point in the current time and selectively based on the three-dimensional spatial position of the human detection point in the historical time comprises: determining whether there is a missing detection point in the current time based on the three-dimensional spatial position of the human detection point in the current time; in response to determining that there is no missing detection point in the current time, determining the three-dimensional spatial position of the human detection point in the current time as the three-dimensional spatial position of each human key point in the current time's human skeleton topology; and in response to determining that there is a missing detection point in the current time, determining the three-dimensional spatial position of the missing detection point in the current time based on the sum of the three-dimensional spatial position of the human detection point in the current time and the three-dimensional spatial position of the human detection point in the historical time as the three-dimensional spatial position of each human key point in the current time's human skeleton topology.

[0010] In an embodiment of the method for controlling the robot arm according to the present application or any of the above embodiments, wherein the three-dimensional spatial position of the human detection point in the historical time comprises a three-dimensional spatial position of a first human detection point and a three-dimensional spatial position of a second human detection point, wherein the three-dimensional spatial position of the first human detection point is the three-dimensional spatial position of the missing detection point determined in the historical time, and the three-dimensional spatial position of the second human detection point is the three-dimensional spatial position of the human key point closest to the missing detection point in the human skeleton topology determined in the historical time.

[0011] In an embodiment of the method for controlling the robot arm according to the present application or any of the above embodiments, wherein the three-dimensional spatial position of the missing detection point in the current time is determined by: determining direction information and bone length information from the three-dimensional spatial position of the second human detection point to the three-dimensional spatial position of the first human detection point; determining a linear interpolation amount based on the direction information and the bone length information; and determining the three-dimensional spatial position of the missing detection point in the current time based on the linear interpolation amount and the three-dimensional spatial position of the human key point closest to the missing detection point in the human skeleton topology in the current time.

[0012] According to the method of controlling the robot arm in any of the embodiments of the present application, the step of controlling the operation of the robot arm based on the distance comprises: determining whether the distance is less than a safety distance threshold; in response to determining that the distance is less than the safety distance threshold, controlling the robot arm to stop operating; and in response to determining that the distance is greater than or equal to the safety distance threshold, controlling the robot arm to keep operating.

[0013] According to a second aspect of the present application, there is provided a system for controlling a robot arm, the system comprising: a memory; a processor coupled to the memory; and a computer program stored on the memory and running on the processor, execution of the computer program causing performance of the steps of the method of controlling a robot arm according to the first aspect of the present application.

[0014] According to a third aspect of the present application, there is provided a computer storage medium comprising instructions which, when executed, perform the steps of the method of controlling a robot arm according to the first aspect of the present application.

[0015] The scheme of controlling a robot arm according to one or more embodiments of the present application can accurately match the human body detection points under different camera perspectives to the same target individual by processing images of a plurality of robot arm operating areas collected by a plurality of cameras to obtain two-dimensional spatial positions of a plurality of human body detection areas and human body detection points, dividing the plurality of human body detection areas into a set of human body detection areas belonging to the same human body object based on feature information extracted from the plurality of human body detection areas, determining three-dimensional spatial positions of human body key points in a human body skeletal topology at a current time based on three-dimensional spatial positions of human body detection points at the current time determined based on the two-dimensional spatial positions of the human body detection points and optionally based on three-dimensional spatial positions of human body detection points at historical times, and accurately estimating three-dimensional spatial positions of missing detection points that can exist at the current time, thereby enabling safe and effective control of the robot arm in a complex human-robot collaboration scenario, and significantly improving the safety and reliability of human-robot collaboration. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and / or other aspects and advantages of the present application will become more apparent and more readily appreciated by referring to the following detailed description in conjunction with the accompanying drawings, in which like reference notations are used to designate and identify similar, equivalent or identical elements. In the drawings:

[0017] Figure 1 A flowchart of a method of controlling a robot arm according to one or more embodiments of the present application is shown.

[0018] Figure 2 A schematic diagram of an arrangement of a plurality of cameras according to one embodiment of the present application is shown.

[0019] Figure 3 A schematic diagram is shown illustrating human object matching of images of multiple robotic arm operating areas captured by multiple cameras, according to an embodiment of this application.

[0020] Figure 4 A schematic block diagram of a system for controlling a robotic arm according to one or more embodiments of this application is shown. Detailed Implementation

[0021] The following detailed description is merely exemplary in nature and is not intended to limit the disclosed technology or its application and use. Furthermore, it is not intended to be bound by any express or implied theory presented in the foregoing technical fields, background art, or the following detailed description.

[0022] In the following detailed description of the embodiments, numerous specific details are set forth in order to provide a more thorough understanding of the disclosed technology. However, it will be apparent to those skilled in the art that the disclosed technology can be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0023] Terms such as "comprising" and "including" indicate that, in addition to the units and steps that are directly and explicitly described in the specification, the technical solution of this application does not exclude the presence of other units and steps that are not directly or explicitly described. Terms such as "first" and "second" do not indicate the order of the units in terms of time, space, size, etc., but are merely used to distinguish the units.

[0024] In the following, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings.

[0025] Figure 1 A flowchart illustrating a method for controlling a robotic arm according to one or more embodiments of this application is shown.

[0026] like Figure 1 As shown, in step S101, images of multiple robotic arm operation areas acquired by multiple cameras at the current moment are processed to obtain multiple human detection areas and the two-dimensional spatial position of human detection points in each human detection area.

[0027] Optionally, in step S101, images of multiple robotic arm operating areas acquired by multiple cameras at the current moment can be input into the pose estimation network model to obtain multiple human detection areas and the two-dimensional spatial positions of human detection points within each human detection area. It should be noted that the robotic arm operating area refers to the set of all positions that the end effector of the robotic arm can reach in space.

[0028] In an embodiment, the pose estimation network model can be implemented as various pose estimation network models such as YOLOv11-Pose, OpenPose, AlphaPose, YOLOv7-Pose, etc. In an embodiment, the two-dimensional spatial position of the human detection point in each human detection region can include part or all of the two-dimensional coordinates (x, y) of 17 standard joint nodes of the human body, which can include nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0029] In an embodiment, the multiple cameras have overlapping regions between their fields of view, for example, using the cross-view arrangement shown in FIG. 2, so that the viewing angles of the respective cameras can be complementary, thereby achieving wide-area coverage of the robot operating region to enhance the stereoscopic perception capability. By introducing overlapping regions between the fields of view of the multiple cameras, the detection robustness and three-dimensional positioning accuracy of the robot and target objects in its operating environment can be improved. Figure 3

[0030] In an embodiment, the YOLOv11-Pose model can be independently run for each camera, for example, the image of the robot operating region captured by camera 1 is input to YOLOv11-Pose model 1, the image of the robot operating region captured by camera 2 is input to YOLOv11-Pose model 2, and the image of the robot operating region captured by camera 3 is input to YOLOv11-Pose model 3, thereby extracting human detection points and human detection regions in real time and synchronously from the images captured by the respective viewing angle cameras, for example, extracting human detection points and human detection regions in real time and synchronously from the images of multiple robot operating regions captured by the three cameras at the current time, thereby providing two-dimensional data support for subsequent three-dimensional reconstruction of the human detection points.

[0031] In step S103, multiple feature information is extracted from the multiple human detection regions, and the multiple human detection regions are divided into one or more human detection region sets based on the extracted multiple feature information, wherein each human detection region set belongs to the same human object and includes multiple human detection regions.

[0032] Optionally, in step S103, the multiple human detection regions can be input to an image classification network to extract multiple feature information, the multiple feature information can be clustered to obtain feature information of one or more clusters, and the multiple human detection regions can be divided into one or more human detection region sets based on the feature information of the one or more clusters.

[0033] ​In one embodiment, the plurality of human detection regions can be input into a pre-trained image classification network to extract a plurality of global high-dimensional feature information, and the extracted plurality of global high-dimensional feature information is input into a density clustering algorithm DBSCAN to obtain global high-dimensional feature information of one or more clusters. In one embodiment, the image classification network can be implemented as a ResNet network, a DenseNet network, an EfficientNet network, a Vision Transformer network, etc. Illustratively, the following formula (1) represents inputting the plurality of human detection regions into a pre-trained ResNet50 classification network to extract a plurality of high-dimensional feature vectors:

[0034]

[0035] wherein I i represents the i-th human detection region, f i represents a d-dimensional feature vector extracted from the i-th human detection region.

[0036] In one embodiment, in the density clustering algorithm DBSCAN, the minimum neighborhood sample number parameter MinPts can be set to 2, that is, in the actual clustering process, for any feature vector f i , if it contains at least 2 feature points in the neighborhood of the preset neighborhood radius ∩, the feature vector f i is regarded as a core point and the feature points in its neighborhood are clustered into the cluster where the core point is located; otherwise, the feature vector f i is judged as an anomaly detection and discarded. By setting the minimum neighborhood sample number parameter MinPts to 2, the feature information of each cluster is ensured to be obtained from the human detection regions of the robot operating area collected by at least two cameras from different perspectives, which is beneficial to subsequent determination of the three-dimensional spatial position of the human detection point.

[0037] In one embodiment, in the density clustering algorithm DBSCAN, the Euclidean distance between any two feature vectors can be determined and compared with a preset neighborhood radius ∈ to determine whether the two feature vectors belong to the same local density region. By using the density clustering algorithm DBSCAN, the density distribution of the plurality of feature information extracted from the plurality of human body detection regions in the feature space can be evaluated to automatically identify and cluster the plurality of feature information of the same human body object in different camera perspectives without specifying the number of clusters, and the robustness is high. For example, the minimum error method can be used to preset the neighborhood radius ∈, for example, 10 neighborhood radii ∈ are selected in the range of 0.1 to 1, the density clustering algorithm DBSCAN is used for clustering processing of the feature information for each selected neighborhood radius ∈ value, the error function value is calculated for each clustering result, and the neighborhood radius ∈ value that minimizes the error function value is selected as the preset neighborhood radius ∈.

[0038] In step S105, the three-dimensional spatial position of the human body detection point at the current moment is determined based on the three-dimensional spatial position of the human body detection point at the current moment determined by the two-dimensional spatial position of the human body detection point in each human body detection region in the set of human body detection regions, and optionally based on the three-dimensional spatial position of the human body detection point at the historical moment.

[0039] Optionally, in step S105, a plurality of first space rays can be determined based on the two-dimensional spatial position of the human body detection point in each human body detection region in the set of human body detection regions, the plurality of first space rays are transformed from the camera coordinate system to the robot base coordinate system to obtain a plurality of second space rays, and the three-dimensional spatial point with the minimum sum of distances to each second space ray is determined in the robot base coordinate system, and the position of the three-dimensional spatial point is determined as the three-dimensional spatial position of the human body detection point at the current moment.

[0040] In one embodiment, the method of hand-eye calibration can be used to jointly calibrate the plurality of cameras and the robot to obtain the homogeneous transformation matrix from the camera coordinate system to the robot base coordinate system, so as to transform the plurality of first space rays from the camera coordinate system to the robot base coordinate system to obtain a plurality of second space rays by using the homogeneous transformation matrix from the camera coordinate system to the robot base coordinate system.

[0041] As an example, the absolute poses of the robot end at positions i and i+1 in the robot end coordinate system E can be obtained from the robot controller and Based on the absolute poses and the homogeneous transformation matrix of the robot end from position i to position i+1 in the robot end coordinate system E is determined where i = 1, 2, …, 10. As an example, the checkerboard calibration board can be moved to ten spatial positions in sequence by the end effector of the robot arm, and images of the checkerboard calibration board at each position are synchronously captured by each camera, and based on the captured images, the absolute poses of the checkerboard calibration board at position i and position i+1 in the camera coordinate system C are determined and based on the absolute poses and determine the homogeneous transformation matrix of the checkerboard calibration board from position i to position i+1 in the camera coordinate system C where i = 1, 2, …, 10.

[0042] As an example, by means of the hand-eye calibration equation AX = XB, let the homogeneous transformation matrix of the camera coordinate system C to the end effector coordinate system E is solved by the least square method As an example, the homogeneous transformation matrix of the end effector coordinate system E to the robot base coordinate system B can be determined by the following equations (2)-(5)

[0043]

[0044] where (x, y, z) represents the translation coordinates of the end effector, (r x , r y , r z ) represents the rotation vector of the end effector, and I represents a 3x3 unit matrix. Thus, the homogeneous transformation matrix of the camera coordinate system C to the robot base coordinate system B can be determined

[0045] As an example, after obtaining the two-dimensional spatial position [u, v] of the human detection point x i in the human detection region in the image of the robot operating area captured by the i-th camera, the human detection point x i is determined as a ray starting from the camera optical center and having a direction , and then the ray can be transformed to the robot base coordinate system by using the homogeneous transformation matrix of the camera coordinate system C i of the i-th camera to the robot base coordinate system B to obtain the ray Y i in the robot base coordinate system. As an example, the homogeneous transformation matrix of the camera coordinate system C i of the i-th camera to the robot base coordinate system B can be represented by the following equation (6):

[0046]

[0047] wherein, represents a rotation matrix, represents a translation vector.

[0048] As an example, the camera coordinate system C i of the i-th camera represented by the above formula (6) can be transformed to the base coordinate system B of the robot arm by the following formula (7): i

[0049]

[0050] As an example, the three-dimensional space point i with the smallest sum of squares of distances to the rays Y

[0051]

[0052] wherein, n represents the number of cameras, represents a spatial ray determined by the two-dimensional space position of the human detection point x i acquired by the i-th camera, represents the three-dimensional space position of the human detection point x i in the base coordinate system of the robot arm.

[0053] Optionally, in step S105, whether there is a missing detection point at the current time can be determined based on the three-dimensional space position of the human detection point at the current time, when it is determined that there is no missing detection point, the three-dimensional space position of the human detection point at the current time can be determined as the three-dimensional space position of each human key point in the human skeletal topology at the current time; when it is determined that there is a missing detection point, the sum of the three-dimensional space position of the missing detection point at the current time determined by the three-dimensional space position of the human detection point at the historical time and the three-dimensional space position of the human detection point at the current time can be determined as the three-dimensional space position of each human key point in the human skeletal topology at the current time. In an embodiment, the human detection point at the current time can be compared with the human key points in the human skeletal topology, if the human detection point at the current time respectively corresponds to the human key points in the human skeletal topology, it is determined that there is no missing detection point at the current time, otherwise it is determined that there is a missing detection point at the current time.

[0054] ​​​In one embodiment, when it is determined that there is a missing detection point, the three-dimensional spatial position of the missing detection point determined at a historical time and the three-dimensional spatial position of the human body key point closest to the missing detection point in the human body skeletal topology determined at the historical time can be obtained, and the three-dimensional spatial position of the missing detection point at the current time can be estimated based on the three-dimensional spatial position of the missing detection point determined at the historical time, the three-dimensional spatial position of the human body key point closest to the missing detection point in the human body skeletal topology determined at the historical time, and the three-dimensional spatial position of the human body key point closest to the missing detection point in the human body skeletal topology at the current time. For example, assuming that the missing detection point at the current time is the head joint, it can be determined that the joint closest to the head joint in the human body skeletal topology is the neck joint, the three-dimensional spatial positions of the head joint and the neck joint determined at the historical time can be obtained, and the three-dimensional spatial position of the head joint at the current time can be estimated based on the three-dimensional spatial positions of the head joint and the neck joint determined at the historical time and the three-dimensional spatial position of the neck joint at the current time.

[0055] In one embodiment, the direction information and the bone length information from the three-dimensional spatial position of the human body key point closest to the missing detection point in the human body skeletal topology determined at the historical time to the three-dimensional spatial position of the missing detection point determined at the historical time can be determined, the linear interpolation amount can be determined based on the direction information and the bone length information, and the three-dimensional spatial position of the missing detection point at the current time can be determined based on the linear interpolation amount and the three-dimensional spatial position of the human body key point closest to the missing detection point in the human body skeletal topology at the current time. For example, the three-dimensional spatial position of the missing detection point at the current time can be estimated by the following formulas (9)-(10)

[0056]

[0057] wherein, represents the three-dimensional spatial position of the missing detection point determined at the historical time, represents the three-dimensional spatial position of the human body key point closest to the missing detection point in the human body skeletal topology determined at the historical time, represents the three-dimensional spatial position of the human body key point closest to the missing detection point in the human body skeletal topology at the current time, and L represents the bone length information from to

[0058] ​In step S107, the distance between each human body key point and the robotic arm key point is determined based on the three-dimensional spatial position of each human body key point in the current human skeleton topology and the three-dimensional spatial position of the robotic arm key point, and the operation of the robotic arm is controlled based on the distance.

[0059] Optionally, in step S107, the minimum distance between each human key point and the robotic arm key point can be determined based on the three-dimensional spatial positions of each human key point in the current human skeletal topology and the three-dimensional spatial positions of the robotic arm key points. It is then determined whether this minimum distance is less than a safety distance threshold. If the minimum distance is less than the safety distance threshold, the robotic arm can be controlled to stop operating; if the minimum distance is greater than or equal to the safety distance threshold, the robotic arm can be controlled to continue operating. As an example, the three-dimensional spatial positions of the human key points in the current human skeletal topology can be determined as follows: Simultaneously, the current three-dimensional spatial position P of the key points of the robotic arm is obtained from the robotic arm controller. robot =[x r ,y r ,z r Then, the minimum distance d between each human body keypoint and the robotic arm keypoint can be determined using the following formula (11). min :

[0060]

[0061] It should be noted that, Figure 1 Steps S105 and S107 shown are performed on a set of human detection regions belonging to the same human object. This enables the three-dimensional reconstruction of each key point of the same human body based on images of the robotic arm's operating area from different camera perspectives, thereby achieving subsequent safe control of the robotic arm.

[0062] The method for controlling a robotic arm according to one aspect of this application processes images of multiple robotic arm operation areas acquired by multiple cameras to obtain the two-dimensional spatial positions of multiple human detection regions and human detection points. Based on the feature information extracted from the multiple human detection regions, the multiple human detection regions are divided into a set of human detection regions belonging to the same human object. This enables accurate matching of human detection points from different camera perspectives to the same target individual. By determining the three-dimensional spatial position of the human detection points at the current moment based on the two-dimensional spatial position of the human detection points, and selectively determining the three-dimensional spatial position of each human key point in the human skeletal topology at the current moment based on the three-dimensional spatial position of the human detection points at historical moments, the method achieves accurate estimation of the three-dimensional spatial position of possible missing detection points at the current moment. This enables safe and effective control of the robotic arm in complex human-machine collaboration scenarios, significantly improving the safety and reliability of human-machine collaboration.

[0063] The following will combine Figure 2 and Figure 3 Further description includes a schematic diagram of the arrangement of multiple cameras according to one or more embodiments of this application and a schematic diagram of cross-field matching of images of multiple robotic arm operating areas acquired by the multiple cameras.

[0064] Figure 2 A schematic diagram of the arrangement of a plurality of cameras according to one embodiment of this application is shown.

[0065] like Figure 2 As shown in the diagram, arrangement 200 illustrates a schematic arrangement of cameras 221, 222, and 223 relative to the robotic arm 210. The fields of view of cameras 221, 222, and 223 overlap, allowing their perspectives to complement each other and thus achieving wide-area coverage of the robotic arm's operating area to enhance stereoscopic perception. Schematably, camera 221 can be positioned on the left side of the robotic arm's operating area, covering the left side and part of the central area; camera 222 can be positioned in the center of the robotic arm's operating area, covering the central area; and camera 223 can be positioned on the right side of the robotic arm's operating area, covering the right side and part of the central area.

[0066] It should be noted that, Figure 2 The schematic arrangement of cameras 221, 222, and 223 relative to robotic arm 210 is shown only as an example. The number and arrangement of cameras may be changed without departing from the spirit and scope of this application.

[0067] Figure 3A schematic diagram is shown illustrating human object matching of images of multiple robotic arm operating areas captured by multiple cameras, according to an embodiment of this application.

[0068] exist Figure 3 In the illustrative matching process 300 shown, image 3101 represents an image of the robotic arm operating area captured by camera 1 at the current moment, image 3102 represents an image of the robotic arm operating area captured by camera 2 at the current moment, and image 3103 represents an image of the robotic arm operating area captured by camera 3 at the current moment, wherein cameras 1, 2 and 3 can be arranged in an intersecting field of view.

[0069] After acquiring images 3101, 3102, and 3103, the acquired images 3101, 3102, and 3103 can be input into a pose estimation network model to extract human detection regions 3201, 3202, and 3203 from images 3101, 3102, and 3103, respectively. Next, the human detection regions 3201, 3202, and 3203 can be input into an image classification network to extract feature information 3301, 3302, and 3303 from the human detection regions 3201, 3202, and 3203.

[0070] After extracting feature information 3301, 3302, and 3303, clustering can be performed on feature information 3301, 3302, and 3303 to obtain feature information 3304, 3305, and 3306 for multiple clusters. Finally, based on the feature information 3304, 3305, and 3306 for multiple clusters, multiple human detection regions 3201, 3202, and 3203 can be divided into multiple human detection region sets 3401, 3402, and 3403, where human detection region set 3401 belongs to human object 1, human detection region set 3402 belongs to human object 2, and human detection region set 3403 belongs to human object 3. In one embodiment, the density clustering algorithm DBSCAN can be used to cluster feature information 3301, 3302, and 3303, where the minimum neighborhood sample number parameter MinPts can be set to 2. This allows feature information 3304 and its corresponding human detection region set 3401 to be identified as anomalies and discarded.

[0071] pass Figure 3 The illustrative matching process 300 shown can detect multiple human detection regions in the images of multiple robotic arm operation areas captured at the current moment by multiple cameras arranged in the form of cross-view, and match multiple human detection regions to different human objects, thereby realizing the accurate detection of the two-dimensional spatial position of the human detection point of the same human object and the reconstruction of the three-dimensional spatial position.

[0072] Figure 4A schematic block diagram of a system for controlling a robotic arm according to one or more embodiments of this application is shown.

[0073] like Figure 4 As shown, the system 400 for controlling the robotic arm includes a memory 410, a processor 420, and a computer program 430 stored in the memory 410 and executable on the processor 420. The processor 420 executes the computer program 430 to implement a method for controlling the robotic arm according to one aspect of this application.

[0074] Alternatively, this application can also be implemented as a computer storage medium storing a program for causing a computer to execute a method for controlling a robotic arm according to one aspect of this application.

[0075] Here, computer storage media can be various types, such as disks (e.g., hard disks, optical disks, etc.), cards (e.g., memory cards, optical cards, etc.), semiconductor memory (e.g., ROM, non-volatile memory, etc.), and tapes (e.g., magnetic tape, cassette tape, etc.).

[0076] Where applicable, the various embodiments provided in this application may be implemented using hardware, software, or a combination of hardware and software. Furthermore, where applicable, without departing from the scope of this application, the various hardware and / or software components described herein may be combined into composite components comprising software, hardware, and / or both. Where applicable, without departing from the scope of this application, the various hardware and / or software components described herein may be divided into sub-components comprising software, hardware, or both. Additionally, where applicable, it is contemplated that software components may be implemented as hardware components, and vice versa.

[0077] The software (such as program code and / or data) according to this application can be stored on one or more computer storage media. It is also contemplated that the software identified herein can be implemented using one or more networked and / or otherwise general-purpose or special-purpose computers and / or computer systems. Where applicable, the order of the various steps described herein can be changed, combined into compound steps, and / or divided into sub-steps to provide the features described herein.

[0078] The embodiments and examples presented herein are provided to best illustrate embodiments of this application and its particular applications, thereby enabling those skilled in the art to implement and use this application. However, those skilled in the art will understand that the above description and examples are provided for ease of illustration and example only. The descriptions presented are not intended to cover all aspects of this application or to limit this application to the precise forms disclosed.

Claims

1. A method of controlling a robot arm, characterized by, The method comprises the following steps: processing images of a plurality of robot operating areas captured by a plurality of cameras at a current time to obtain a plurality of human detection regions and two-dimensional spatial positions of human detection points in each human detection region; extracting a plurality of feature information from the plurality of human detection regions and dividing the plurality of human detection regions into one or more human detection region sets based on the extracted plurality of feature information, wherein each human detection region set belongs to the same human object and comprises a plurality of human detection regions; determining three-dimensional spatial positions of human detection points at the current time based on the two-dimensional spatial positions of the human detection points in the plurality of human detection regions in each human detection region set, and optionally based on three-dimensional spatial positions of human detection points at a historical time, three-dimensional spatial positions of each human key point in a human skeleton topology at the current time; and determining distances between each human key point and a robot key point based on the three-dimensional spatial positions of each human key point in the human skeleton topology at the current time and the three-dimensional spatial position of the robot key point, and controlling operation of the robot based on the distances.

2. The method of claim 1, wherein the plurality of cameras have overlapping regions between fields of view, and processing images of a plurality of robot operating areas captured by a plurality of cameras at a current time to obtain a plurality of human detection regions and two-dimensional spatial positions of human detection points in each human detection region comprises: inputting the images of the plurality of robot operating areas to a pose estimation network model to obtain the plurality of human detection regions and the two-dimensional spatial positions of human detection points in each human detection region.

3. The method of claim 1, wherein extracting a plurality of feature information from the plurality of human detection regions and dividing the plurality of human detection regions into one or more human detection region sets based on the extracted plurality of feature information comprises: inputting the plurality of human detection regions to an image classification network to extract the plurality of feature information; performing clustering processing on the plurality of feature information to obtain feature information of one or more clusters; and dividing the plurality of human detection regions into one or more human detection region sets based on the feature information of the one or more clusters.

4. The method of claim 1, wherein the three-dimensional spatial positions of human detection points at the current time are determined by: determining a plurality of first spatial rays based on the two-dimensional spatial positions of the human detection points in the plurality of human detection regions in each human detection region set; transforming the plurality of first spatial rays from a camera coordinate system to a robot base coordinate system to obtain a plurality of second spatial rays; and determining a three-dimensional spatial point with a minimum sum of squares of distances to each second spatial ray in the robot base coordinate system, and determining the position of the three-dimensional spatial point as the three-dimensional spatial position of the human detection point at the current time. 5.The method of claim 1, wherein determining the three-dimensional spatial position of each human key point in the current time’s human skeleton topology based on the three-dimensional spatial position of the human detection point determined by the two-dimensional spatial position of the human detection point within the plurality of human detection regions in the set of human detection regions and selectively based on the three-dimensional spatial position of the human detection point at a historical time comprises: determining whether the current time’s human detection point exists a missing detection point based on the three-dimensional spatial position of the current time’s human detection point; in response to determining that the current time’s human detection point does not exist the missing detection point, determining the three-dimensional spatial position of the current time’s human detection point as the three-dimensional spatial position of each human key point in the current time’s human skeleton topology; and in response to determining that the current time’s human detection point exists the missing detection point, determining the three-dimensional spatial position of the current time’s missing detection point as a sum of the three-dimensional spatial position of the current time’s human detection point and the three-dimensional spatial position of the human detection point at the historical time determined by the three-dimensional spatial position of the human detection point within the plurality of human detection regions in the set of human detection regions. 6.The method of claim 5, wherein the three-dimensional spatial position of the human detection point at the historical time comprises a first human detection point’s three-dimensional spatial position and a second human detection point’s three-dimensional spatial position, wherein the first human detection point’s three-dimensional spatial position is the three-dimensional spatial position of the missing detection point determined at the historical time, and the second human detection point’s three-dimensional spatial position is the three-dimensional spatial position of the human key point closest to the missing detection point in the human skeleton topology determined at the historical time. 7.The method of claim 6, wherein the three-dimensional spatial position of the current time’s missing detection point is determined by: determining direction information and bone length information from the second human detection point’s three-dimensional spatial position to the first human detection point’s three-dimensional spatial position; determining a linear interpolation amount based on the direction information and the bone length information; and determining the three-dimensional spatial position of the current time’s missing detection point based on the linear interpolation amount and the three-dimensional spatial position of the human key point closest to the missing detection point in the human skeleton topology at the current time. 8.The method of claim 1, wherein controlling the operation of the robot arm based on the distance comprises: determining whether the distance is less than a safety distance threshold; in response to determining that the distance is less than the safety distance threshold, controlling the robot arm to stop operating; and in response to determining that the distance is greater than or equal to the safety distance threshold, controlling the robot arm to keep operating. The system comprises: a memory; a processor coupled with the memory; and a computer program stored on the memory and running on the processor, execution of the computer program causing performance of the method of controlling a robot arm according to any one of claims 1-8. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 9. A system for controlling a robotic arm, the system comprising: ​ ​ ​ ​ 10. A computer storage medium, characterized in that, The computer storage medium comprises instructions which, when executed, perform the method of controlling a robot arm according to any one of claims 1-8. The computer storage medium comprises instructions which, when executed, perform the method of controlling a robot arm according to any one of claims 1-8.

Images