Obstacle detection method, electronic equipment and storage medium

Through multiple image acquisition devices, point cloud data is collected and coordinate conversion and filtering is performed to identify and classify obstacles, solving the problems of inaccurate and high cost of obstacle detection in the prior art, and achieving the accuracy of obstacles in industrial scenarios Perception and security detection.

CN119068326APending Publication Date: 2024-12-03SHANGHAI AIRCRAFT MFG
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202310634744.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing obstacle detection methods have problems of inaccurate detection and high cost in industrial scenarios, and it is impossible to achieve accurate environmental perception and hierarchical perception.

Method used

By acquiring the point cloud data collected by at least two image acquisition devices, and combining the camera to base coordinate conversion relationship determines alternative point cloud data, filters under the same coordinate system to obtain target point cloud data, and performs obstacle identification and grading It means that obstacles are graded in combination with obstacle distance and grading strategies.

Benefits of technology

It realizes that without high-level modeling, data is comprehensively collected and processed, the accuracy and efficiency of obstacle detection are improved, data omissions are reduced, and obstacles of any distance can be accurately detected, and the safety of collision detection is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068326A_ABST
    Figure CN119068326A_ABST
Patent Text Reader

Abstract

The invention discloses an obstacle detection method, electronic equipment and a storage medium. The method comprises the steps that point cloud data acquired by at least two image acquisition devices are acquired, alternative point cloud data are determined according to the point cloud data in combination with a coordinate conversion relation from a corresponding camera to a base, and the alternative point cloud data are in the same coordinate system; filtering each alternative point cloud data to obtain target point cloud data; obstacle identification is carried out according to the cloud data of each target point, and obstacle information is determined; the obstacle distance between the obstacle and the mechanical arm is determined according to the information of each obstacle, and the obstacle is represented in a graded manner according to the obstacle distance and the obstacle grading strategy, so that the problems of inaccurate obstacle detection and high detection cost are solved, the accuracy of obstacle detection is improved, the data processing amount is reduced, and the detection efficiency is improved. And the data processing efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an obstacle detection method, electronic equipment and storage medium. Background Art

[0002] In recent years, with the continuous expansion of robot application scenarios and the increase in task complexity, new challenges have been raised to the safety of human-machine collaboration scenarios, especially the need for robots to constantly perceive changes in the external environment and have the ability to detect and identify static and dynamic obstacles. In actual industrial scenarios, the movement speed and movement path of obstacles change rapidly. For robots, how to achieve fast, complete and accurate perception of obstacles through external sensors is the key to the robot's execution of established safety strategies, and is of great significance to ensuring the safety of human-machine collaboration scenarios.

[0003] Existing obstacle detection methods include: one is to perform high-precision modeling of the robot's operating environment. This method will increase the amount of calculation for high-precision reconstruction of the entire industrial environment, resulting in high computing costs; it can also be based on a variety of sensing technologies to track the work of humans and robots in the same environment, specifically using ultrasonic and infrared proximity sensors to detect the distance between humans and robots, but this method cannot provide accurate environmental perception information, and the above methods cannot achieve hierarchical perception. Summary of the invention

[0004] The present invention provides an obstacle detection method, an electronic device and a storage medium to solve the problems of inaccurate obstacle detection and high detection cost.

[0005] According to one aspect of the present invention, there is provided an obstacle detection method, comprising:

[0006] Acquire point cloud data collected by at least two image acquisition devices, and determine candidate point cloud data according to each of the point cloud data combined with a corresponding camera-to-base coordinate transformation relationship, wherein each of the candidate point cloud data is in the same coordinate system;

[0007] Filtering each of the candidate point cloud data to obtain target point cloud data;

[0008] Perform obstacle identification according to each of the target point cloud data to determine obstacle information;

[0009] The obstacle distance between the obstacle and the robot arm is determined according to the obstacle information, and the obstacles are graded and represented according to the obstacle distances combined with the obstacle classification strategy.

[0010] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0011] at least one processor; and

[0012] a memory communicatively connected to the at least one processor; wherein,

[0013] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the obstacle detection method described in any embodiment of the present invention.

[0014] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the obstacle detection method described in any embodiment of the present invention when executed.

[0015] The technical solution of the embodiment of the present invention obtains point cloud data collected by at least two image acquisition devices, determines candidate point cloud data according to each of the point cloud data combined with the corresponding camera to base coordinate transformation relationship, and each of the candidate point cloud data is in the same coordinate system; filters each of the candidate point cloud data to obtain target point cloud data; identifies obstacles according to each of the target point cloud data to determine obstacle information; determines the obstacle distance between the obstacle and the robotic arm according to each of the obstacle information, and classifies each of the obstacles according to each of the obstacle distances combined with an obstacle classification strategy, thereby solving the problems of inaccurate obstacle detection and high detection cost. Obstacle detection is realized by collecting point cloud data through multiple image acquisition devices, and comprehensive data collection and operation processing can be realized without high-level modeling, and the image acquisition device can provide accurate perception information by collecting point cloud data. Through the coordinate transformation relationship from the camera to the base, each point cloud data is converted to the same coordinate system to obtain the alternative point cloud data, which can accurately collect all the data in the space, reduce the errors caused by mutual occlusion of objects, effectively avoid data omissions, improve the accuracy of obstacle detection, and realize the detection of obstacles at any distance; and through filtering to determine the target point cloud data, obstacle identification is performed according to the target point cloud data to determine the obstacle information, further improve the accuracy of obstacle detection, and can reduce the amount of data processing and improve data processing efficiency; obstacles are graded according to the obstacle distance combined with the obstacle classification strategy, and obstacles in the environment are represented separately according to different representation methods, which is conducive to the staff to effectively distinguish obstacles at different distances and improve the safety of collision detection.

[0016] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 is a flow chart of an obstacle detection method provided according to Embodiment 1 of the present invention;

[0019] Figure 2 is a flow chart of an obstacle detection method provided according to Embodiment 2 of the present invention;

[0020] Figure 3a is an example diagram showing the positions of a robotic arm and an image acquisition device provided according to the second embodiment of the present invention;

[0021] Figure 3b This is an example diagram showing the positions of a robotic arm and a multi-image acquisition device provided according to Embodiment 2 of the present invention;

[0022] Figure 4 is an example diagram of an obstacle detection process provided according to Embodiment 2 of the present invention;

[0023] Figure 5 Schematic diagram of the structure of an electronic device for implementing the obstacle detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] Embodiment 1

[0027] Figure 1 A flowchart of an obstacle detection method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of accurately detecting obstacles. The method can be executed by an obstacle detection device. The obstacle detection device can be implemented in the form of hardware and / or software. The obstacle detection device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0028] S101, acquiring point cloud data acquired by at least two image acquisition devices, and determining candidate point cloud data according to each point cloud data in combination with a corresponding camera-to-base coordinate transformation relationship, wherein each candidate point cloud data is in the same coordinate system.

[0029] In this embodiment, the image acquisition device can be a camera, a video recorder, a scanner, an infrared thermal imager, etc. The coordinate conversion relationship from the camera to the base can be understood as the conversion relationship when the coordinates in the camera coordinate system (the coordinate system of the image acquisition device) are converted to the coordinate system of the base of the robot arm; the alternative point cloud data can be specifically understood as point cloud data in a specific coordinate system, which can be directly used for obstacle detection and recognition.

[0030] The image acquisition device in the embodiment of the present application can be set at any position in the workspace. Taking the robot arm working in a confined space as an example, image acquisition devices can be set at different positions in this confined space. The number of image acquisition devices can be determined according to the size of the space, the spatial structure, etc. When setting the image acquisition device in the embodiment of the present application, the position and angle of each image acquisition device can be set according to the needs to ensure that all image acquisition devices can collect all positions in the space together. Each image acquisition device can collect point cloud data at the same frequency and send the collected point cloud data to the execution device. After the execution device obtains each point cloud data, since the coordinate system of each point cloud data is different, the point cloud data collected by each image acquisition device is in its own coordinate system. Therefore, the present application performs coordinate conversion on each point cloud data and unifies it into the same coordinate system. Since the position and angle of each image acquisition device are different, the coordinate conversion relationship from its camera to the base is different. The coordinate transformation relationship between the camera and the base of each image acquisition device is determined in advance. After acquiring the point cloud data, the coordinate transformation relationship between the camera and the base corresponding to the point cloud data is determined. The point cloud data is transformed into the base coordinate system through the coordinate transformation relationship between the camera and the base. The obtained data can be directly used as candidate point cloud data, or the obtained data can be fused to remove the error of the coordinate transformation, and the obtained data can be used as candidate point cloud data. The candidate point cloud data obtained in this step are in the same coordinate system.

[0031] S102: Filter each candidate point cloud data to obtain target point cloud data.

[0032] In this embodiment, the target point cloud data can be specifically understood as the point cloud data obtained after filtering out useless data such as noise. The method of filtering each candidate point cloud data can be downsampling, noise reduction, etc., and the point cloud data obtained by filtering out useless data is the target point cloud data.

[0033] S103: Perform obstacle identification according to each target point cloud data to determine obstacle information.

[0034] In this embodiment, the obstacle information may be information such as the location coordinates and category of the obstacle. In this embodiment of the application, an obstacle recognition algorithm may be set according to requirements to obtain a specific type of obstacle information.

[0035] An obstacle recognition algorithm is pre-set to perform obstacle recognition on each target point cloud data, determine the position of the obstacle, and obtain obstacle information. The obstacle information can be the coordinates of each point in the obstacle, or it can be represented by several data. For example, when the obstacle is a rectangle, the obstacle information can be the center point, length and width.

[0036] S104: Determine the obstacle distance between the obstacle and the robot arm according to the obstacle information, and classify the obstacles according to the obstacle distance and the obstacle classification strategy.

[0037] In this embodiment, the obstacle classification strategy can be specifically understood as a strategy of representing different obstacles in different ways. For example, obstacles that are far away are not represented, and obstacles that are close are represented in a set manner.

[0038] A distance calculation method is pre-set, and the distance between the obstacle and the robotic arm is calculated according to the distance calculation method. When calculating the obstacle distance, the embodiment of the present application can calculate the shortest distance between the obstacle and the robotic arm, or the distance between the center points, etc. An obstacle classification strategy is pre-set, and the corresponding representation method of the obstacle is determined according to the obstacle classification strategy combined with the obstacle distance, and the obstacle is represented according to the corresponding representation method.

[0039] The embodiment of the present invention provides an obstacle detection method, which solves the problems of inaccurate obstacle detection and high detection cost. By collecting point cloud data through multiple image acquisition devices, obstacle detection can be realized, and comprehensive data collection and operation processing can be realized without high-level modeling, and the image acquisition device can provide accurate perception information by collecting point cloud data. Through the coordinate conversion relationship from the camera to the base, each point cloud data is converted to the same coordinate system to obtain the candidate point cloud data, which can accurately collect all data in the space, reduce the error caused by mutual occlusion of objects, effectively avoid data omission, improve the accuracy of obstacle detection, and realize the detection of obstacles at any distance; and through filtering to determine the target point cloud data, obstacle identification is performed according to the target point cloud data to determine the obstacle information, further improve the accuracy of obstacle detection, and reduce the amount of data processing and improve the data processing efficiency; obstacles are graded by combining the obstacle distance with the obstacle classification strategy, and obstacles in the environment are represented respectively according to different representation methods, which is conducive to the staff to effectively distinguish obstacles at different distances and improve the safety of collision detection.

[0040] Embodiment 2

[0041] Figure 2 This is a flow chart of an obstacle detection method provided in Embodiment 2 of the present invention. This embodiment is refined on the basis of the above embodiment. Figure 2 As shown, the method includes:

[0042] S201. Acquire point cloud data acquired by at least two image acquisition devices.

[0043] Exemplarily, the image acquisition device may be a Kinect camera.

[0044] S202. For the point cloud data collected by each image acquisition device, each point cloud data is converted to the base coordinate system through the corresponding camera-to-base coordinate conversion relationship to obtain the corresponding base point cloud data.

[0045] In this embodiment, the base point cloud data can be specifically understood as the coordinate data in the coordinate system of the base of the robot. When processing the point cloud data collected by each image acquisition device, the embodiment of the present application can process the point cloud data collected by the same image acquisition device in sequence. For each image acquisition device, each point cloud data collected by it is respectively transformed through the coordinate transformation relationship from the camera to the base to obtain the corresponding base point cloud data. The coordinate transformation relationship from the camera to the base in the embodiment of the present application can be a matrix. The point cloud data is directly multiplied by the matrix to obtain the base point cloud data.

[0046] As an optional embodiment of this embodiment, this optional embodiment further optimizes the step of determining the coordinate transformation relationship from the camera to the base, including the following steps:

[0047] A1. Determine the posture transformation relationship of the image acquisition device serving as a reference device under two robot arm postures, and determine the first coordinate transformation matrix between the reference device and the robot arm base according to the posture transformation relationship. The first coordinate transformation matrix is ​​the coordinate transformation relationship from the camera corresponding to the image acquisition device serving as the reference device to the base.

[0048] In this embodiment, the reference device can be specifically understood as an image acquisition device that serves as a reference standard for coordinate conversion. This application sets a reference device to determine the coordinate transformation relationship of other image acquisition devices through the reference device. The posture transformation relationship can be specifically understood as the relationship expression between the base, end, calibration plate, and camera of the manipulator when the manipulator transforms different postures; the first coordinate transformation matrix can be specifically understood as the coordinate transformation relationship of the coordinate system of the image acquisition device of the reference device to the coordinate system of the base.

[0049] An image acquisition device is pre-installed and used as a reference device. A calibration plate cali1 is fixed to the end of the manipulator to determine the coordinate transformation relationship between the end of the manipulator and the base of the manipulator in each posture. Since the position of the base of the manipulator and the position of the image acquisition device remain unchanged, the position relationship between the base of the manipulator and the image acquisition device is determined. Therefore, the coordinate transformation relationship between the end of the manipulator and the base of the manipulator in the two postures is the same. An equation is established to obtain the posture transformation relationship. By calculating the posture transformation relationship, a first coordinate transformation matrix is ​​obtained. The first transformation matrix is ​​the coordinate transformation relationship from the camera to the base corresponding to the image acquisition device as a reference device.

[0050] In the embodiment of the present application, the coordinate conversion relationship between the camera and the base can be determined during the calibration process. During calibration, a calibration scene is set up, a calibration plate cali1 is fixed to the end of the robot arm, a calibration plate cali2 is fixed in the base coordinate system of the robot arm, and an image acquisition device cam is fixed outside the robot arm to ensure that the image acquisition device cam can capture two calibration plates at the same time. For example, Figure 3a An example diagram showing the positions of a robotic arm and an image acquisition device is provided. Figure 3a The image acquisition device 31 in the figure is the reference device, the end 32 of the robot arm is fixedly connected to the calibration plate cali1, and the calibration plate cali2 is fixed in the base coordinate system of the robot arm; the posture of the robot arm is changed to ensure that the end position of the robot arm remains unchanged. The posture transformation relationship of the robot arm before and after is as follows:

[0051]

[0052] in, is the coordinate transformation relationship from the end of the manipulator to the base of the manipulator at the position of the manipulator at time T1; is the coordinate transformation relationship from the base of the manipulator to the image acquisition device in the position of the manipulator at time T1; is the coordinate transformation relationship from the image acquisition device to the calibration plate cali1 in the position of the robot arm at time T1; The coordinate transformation relationship from the end of the robot arm to the base of the robot arm at the position of the robot arm at time T2; The coordinate transformation relationship from the base of the robot arm to the image acquisition device at the position of the robot arm at time T2; The coordinate transformation relationship from the image acquisition device to the calibration plate cali1 in the position of the robot at time T2.

[0053] In the above expression, and is unknown, and since the positions of the robot base and the image acquisition device have not changed, and The rest are known and can be determined in different ways. The above formula can be transformed into the following formula:

[0054]

[0055] Coordinate transformation from the end of the robot arm to the base of the robot arm and The coordinate transformation from the image acquisition device cam to the calibration plate cali1 can be calculated by the mechanical arm structure size and the rotation angle of each joint and The camera projection imaging relationship can be used for calculation, and the equation can be converted into a linear equation for solution and calculation through Lie group and Lie algebra. and The above calculation method is obtained (Right now

[0056] or ) is the coordinate transformation relationship from the base to the camera, and then the coordinate transformation relationship from the camera to the base is obtained

[0057] A2. For each image acquisition device to be converted, the second coordinate transformation matrix from the first calibration plate to the base of the robot is determined based on the coordinate transformation relationship from the first calibration plate to the reference device and the first coordinate transformation matrix. The first calibration plate is fixed in the base coordinate system of the robot.

[0058] In this embodiment, the device to be converted can be specifically understood as an image acquisition device that has the requirement to be aligned with the coordinate system of the reference device; the first calibration plate is a calibration plate fixed in the base coordinate system of the robot arm, namely cali2; the second coordinate transformation matrix can be specifically understood as the coordinate transformation relationship of converting the first calibration plate to the coordinates in the base coordinate system.

[0059] After the reference device is determined, the remaining image acquisition devices are used as devices to be converted, and the second coordinate transformation matrix of the device to be converted is determined according to the first coordinate transformation matrix of the reference device. That is, when implementing calibration, the present application can only perform a complex calibration on the reference device once to determine the first coordinate transformation relationship, and the remaining image acquisition devices only need to be calibrated according to the first coordinate transformation relationship, which greatly simplifies the complexity of the camera calibration process and improves the calibration efficiency. The camera calibration in the prior art needs to be repeated, which is very time-consuming and labor-intensive. The embodiments of the present application can greatly simplify the calibration process, save calibration time, and reduce costs.

[0060] For each image acquisition device to be converted, determine the coordinate transformation relationship from the first calibration plate to the reference device. It can be determined by the camera projection imaging relationship; Multiply by the first coordinate transformation matrix Get the coordinate transformation relationship from the first calibration plate to the robot base That is the second coordinate transformation matrix.

[0061] A3. According to the second coordinate transformation matrix and the coordinate transformation relationship between the device to be transformed and the first calibration plate, determine the third coordinate transformation matrix from the device to be transformed to the base of the robotic arm. The third coordinate transformation matrix is ​​the coordinate transformation relationship from the camera corresponding to the image acquisition device of the device to be transformed to the base.

[0062] In this embodiment, the third coordinate transformation matrix can be specifically understood as a coordinate transformation relationship for transforming the coordinates in the coordinate system of the image acquisition device, which is the device to be transformed, into the coordinates in the base coordinate system.

[0063] Calculate the coordinate transformation relationship from the device to be transformed to the first calibration plate based on the camera projection imaging relationship The second coordinate transformation matrix and Multiply them to get the third coordinate transformation matrix from the device to be converted to the robot base

[0064] For example, Figure 3b An example diagram showing the positions of a manipulator and multiple image acquisition devices is provided. Multiple image acquisition devices cam are deployed in the above manner to determine the coordinate transformation relationship from the camera to the base corresponding to the image acquisition device cam of the device to be transformed. In the figure, three image acquisition devices 31 are taken as an example, cam1 can be used as a reference device, cam2 and cam3 are used as devices to be transformed, and cali2 is a first calibration plate fixed in the base coordinate system of the manipulator.

[0065] In the embodiment of the present application, the third coordinate transformation matrix corresponding to all the devices to be converted can be determined according to the above steps A2-A3.

[0066] S203: If the image acquisition device is a reference device, the point cloud data of each base is used as candidate point cloud data.

[0067] In this embodiment, the candidate base point cloud data can be specifically understood as base point cloud data used to determine the candidate point cloud data. If the image acquisition device is a reference device, there is no need to process the base point cloud data corresponding to the image acquisition device, and each base point cloud data is directly used as the candidate base point cloud data.

[0068] S204: If the image acquisition device is a device to be converted, the point cloud data of each base is used as the point cloud data to be processed.

[0069] In this embodiment, the point cloud data to be processed can be specifically understood as point cloud data with processing requirements, which needs to be aligned with the base point cloud data corresponding to the reference device to reduce errors.

[0070] S205, determining a coordinate conversion matrix according to each of the to-be-processed point cloud data in combination with the base point cloud data corresponding to the image acquisition device as a reference device, and converting each of the to-be-processed point cloud data into candidate base point cloud data according to the coordinate conversion matrix.

[0071] In this embodiment, the coordinate conversion matrix can be specifically understood as the transformation matrix used to convert one coordinate to another coordinate. Since there is a certain error when the coordinates in the camera coordinate system are converted to the base coordinate system, taking the two image acquisition devices capturing the same point as an example, the point cloud data of the same point captured by the two image acquisition devices are different, and the base point cloud data obtained by converting them to the base coordinate system may also be different. In fact, the two base point cloud data obtained by conversion should be the same, so it is necessary to eliminate the error by matching and fusion.

[0072] Through point cloud fusion, the point cloud data to be processed and the base point cloud data corresponding to the image acquisition device of the reference device are matched and fused. The matching fusion can use a preset algorithm, such as the ICP algorithm. Through the algorithm, each point cloud data to be processed and the corresponding base point cloud data as the reference device are matched and fused to obtain a coordinate transformation matrix. The coordinate transformation matrix includes a rotation matrix and a translation matrix. The coordinate transformation matrix can align the point cloud data to be processed with the base point cloud data of the reference device. After determining the coordinate transformation matrix, each point cloud data to be processed is multiplied by the coordinate transformation matrix to obtain candidate base point cloud data.

[0073] The embodiment of the present application determines the corresponding coordinate transformation matrix according to the point cloud data to be processed corresponding to each device to be converted, and then implements the coordinate transformation to obtain the candidate base point cloud data.

[0074] Exemplarily, the embodiment of the present application provides a matching fusion process, which fuses the point cloud data through the ICP matching algorithm to solve the coordinate transformation matrix. The specific algorithm process is as follows:

[0075] 1. Initialize the rotation matrix R and the translation matrix T, and take the point cloud data to be processed as input. According to the R and T matrices, P(T) can be calculated, that is, the coordinates of the point cloud data to be processed in the coordinate system of the base point cloud data corresponding to the reference device.

[0076] In this step, R and T are initialized. Initial values ​​can be assigned to R and T. For example, the initial value of R is the unit matrix, and the initial value of T is [0, 0, 0], indicating that the point cloud data to be processed does not need to be rotated or translated.

[0077] 2. Find the point closest to P(T) in the base point cloud data corresponding to the reference device, and the distance is less than the specified threshold.

[0078] 3. Calculate the Euclidean distance error for the points matched in the second step, and optimize the R and T matrices by the least squares method.

[0079] 4. Bring the R and T matrices optimized in the third step back to the first step and iterate again until the iteration meets the requirements to obtain the final optimized R and T matrices. The final optimized R and T matrices are the coordinate transformation matrices.

[0080] The iteration requirements in the embodiment of the present application can set different parameters, namely the maximum number of iterations, the mean square error, the maximum distance between corresponding points, and the minimum conversion difference. If the number of iterations is not exceeded, the iteration requirements can be considered to be met if the other three requirements are met, and the iteration is stopped.

[0081] In the embodiment of the present application, other image acquisition devices use the reference device as a reference standard to perform coordinate transformation. In the embodiment of the present application, since errors may exist in the coordinate transformation of multiple image acquisition devices, an image acquisition device is selected as a reference device to align the base point cloud data corresponding to the other image acquisition devices with the base point cloud data corresponding to this image acquisition device.

[0082] S206: De-duplicate the point cloud data of each candidate base to obtain candidate point cloud data.

[0083] When a point is repeatedly captured by multiple image acquisition devices, multiple coordinate information will appear. Therefore, after data alignment, the repeatedly captured points are deduplicated. The deduplication process in the embodiment of the present application can be implemented using a preset algorithm to deduplicate the point cloud data of each candidate base, retain only one corresponding point cloud data for the repeatedly captured points, and obtain each candidate point cloud data.

[0084] S207: Filter each candidate point cloud data to obtain target point cloud data.

[0085] As an optional embodiment of this embodiment, this optional embodiment further filters each candidate point cloud data to obtain target point cloud data, which is optimized as follows:

[0086] B1. Downsample and denoise the candidate point cloud data to obtain denoised point cloud data.

[0087] In this embodiment, the de-noised point cloud data can be specifically understood as point cloud data obtained after noise is removed. In this embodiment of the application, a preset downsampling algorithm and a noise reduction processing algorithm can be used to downsample and de-noise the candidate point cloud data to obtain the de-noised point cloud data.

[0088] Exemplarily, the present application embodiment provides an alternative point cloud data processing method, calculates the robot workspace, finds the maximum working range of the workspace in the Cartesian coordinate system, and

[0089] The bounding box in the interval [x1,x2],[y1,y2],[z1,z2] indicates that when the candidate point cloud coordinates exceed the above space, the candidate point cloud coordinates are filtered out to reduce the redundant point cloud; further, the downsampled candidate point cloud data is subjected to denoising, and the statistical filter with the best denoising effect is used to calculate a i The average distance from a point to its nearest 10 points. Points with average distances outside the standard range of Gaussian distribution are defined as outliers and removed. Furthermore, the point cloud area is divided into several equal squares, and the average value of the (X, Y, Z) coordinates of all points in each square is calculated, and the average point is used to replace all points in the square to achieve further downsampling. Through the above data processing, the noise-reduced point cloud data is obtained.

[0090] B2. Fit a plane according to each de-noised point cloud data, and determine the largest fitted plane.

[0091] Calculate the normal vector of the denoised point cloud data, estimate the normal vector by plane fitting method, determine the normal vector of each denoised point cloud data, fit the plane according to the normal vector of each denoised point cloud data, compare the sizes of each plane after determining them, and select the largest plane.

[0092] Exemplarily, the present application provides a plane fitting method, comprising the following steps: calculating the normal vector of the de-noised point cloud data, estimating the normal vector using a method based on local surface fitting, searching for the de-noised point cloud data a i The 10 adjacent points {a1, a2…a 10}, calculate the local plane P of 10 points in the least squares sense:

[0093]

[0094] Calculate the unit normal vector n of plane P i , calculate n i and its surrounding normal vector n j If the inner product is negative, the direction needs to be reversed to obtain the normal vector n with a certain direction. i .

[0095] Randomly select the minimum point cloud set a that can be used to calculate the model parameters from the preprocessed denoised point cloud data, and use the minimum point cloud set a to solve the model parameters of the input denoised point cloud data. The model parameters can be used to determine the model parameters of a plane. Determine the number of matching point clouds in the point cloud set that meet the definition of the plane. If the proportion of points that do not conform to the plane model in all outlier point sets is higher than 99%, the model is judged to be valid, and the model parameters are re-estimated using the identified points, and the iteration is terminated. Repeat the above steps 30 times to improve the accuracy of the model iteration. Each plane is obtained in the above manner.

[0096] B3. Filter the noise-reduced point cloud data in the largest plane to obtain the target point cloud data.

[0097] Determine all the noise-reduced point cloud data in the largest plane, filter out this part of the point cloud data, and the remaining is the target point cloud data. The present application determines the largest plane as a static obstacle and filters it out for subsequent dynamic obstacle identification. The largest plane in the embodiment of the present application is usually the workbench of the robot arm.

[0098] The embodiments of the present application can obtain high-precision and high-completeness point cloud data. By performing time synchronization correction, internal parameter correction, external parameter coordinate correction, error correction and multi-point cloud fusion on multiple image acquisition devices, the multi-camera point clouds are aligned with the actual world positions, reducing the errors caused by mutual occlusion of objects, and obtaining multi-camera point cloud data with high matching degree and high accuracy.

[0099] S208: Perform obstacle identification according to each target point cloud data to determine obstacle information.

[0100] As an optional embodiment of this embodiment, this optional embodiment further performs obstacle identification based on each target point cloud data to determine obstacle information and optimizes it as follows:

[0101] C1. Cluster each target point cloud data to obtain clustered point cloud data.

[0102] In this embodiment, clustered point cloud data can be specifically understood as point cloud data obtained after clustering. The target point cloud data belonging to the same object are segmented and clustered into the same point cloud cluster. In this embodiment of the application, a clustering algorithm can be pre-set, and each target point cloud data is clustered by the clustering algorithm to obtain clustered point cloud data.

[0103] Exemplarily, the present application provides a clustering method: the input point cloud set and P are initialized through the KD-tree data structure, and the structure is used for storage and search. In the established KD-tree structure, an empty list C of the cluster set is initialized, and an empty queue Q is initialized, which contains the points to be judged. For each point p i ∈P loop execution:

[0104] 1. Change point p i Join the queue Q;

[0105] 2. For each point p i ∈Q, search point p i Surrounding radius r <d thresh The set of points in the neighborhood sphere For the neighborhood point set Points in If the point has not been processed yet, Join the queue Q;

[0106] 3. If all points in queue Q have been processed, add queue Q to point cloud cluster C and reinitialize the queue to an empty list;

[0107] 4. Repeat the above steps until all points p i ∈P are all processed and divided into different point cloud clusters C to obtain clustered point cloud data.

[0108] It should be noted that the clustered point cloud data obtained in the embodiment of the present application may refer to all point cloud data in a cluster; or it may be a point cloud data, and multiple clustered point cloud data constitute a cluster.

[0109] C2. Perform bounding box representation on each cluster point cloud data to obtain obstacle information of the obstacle.

[0110] A cluster usually corresponds to an obstacle, which can be a person, a table, a toolbox, etc. The bounding box algorithm is used to represent the cluster point cloud data corresponding to each cluster, and the obstacle information of an obstacle is obtained. The obstacle information determined in this step can only include the coordinates of the obstacle, and the obstacle information can be the coordinates of the edge of the obstacle, or the center coordinates of the obstacle, etc.

[0111] Exemplarily, an embodiment of the present application provides a method for representing an obstacle with a bounding box, comprising the following steps:

[0112] 1. Get the three-dimensional coordinates of all clustered point cloud data in the point cloud cluster and store them in an array.

[0113] 2. Solve the covariance matrix of the three-dimensional coordinates.

[0114] 3. Solve for the eigenvector of the covariance matrix and assign the eigenvector to the column vector matrix M.

[0115] 4. Transform the center of the point set to the origin through the translation transformation matrix, multiply it by the M matrix, and perform a rotation transformation.

[0116] 5. Find x for the transformed coordinates max 、x min ,y max ,y min 、z max 、z min And the OBB geometric center and side length.

[0117] 6. Transform the obtained OBB coordinates back to the geometric center of the original point set, that is, multiply by M -1 , and use the inverse of the translation transformation matrix in step 4 to translate the coordinates back to their original positions.

[0118] 7. Return the coordinate axis direction of the OBB, the side length of the bounding box, and the center coordinates of the bounding box to realize the bounding box representation of different obstacles after segmentation.

[0119] S209, obtaining the thermal map, depth map and RGB image collected by each image acquisition device, where the depth map is point cloud data.

[0120] In this embodiment, the heat map, depth map and RGB image can all be acquired by an image acquisition device, and the image acquisition device can be configured with different cameras to respectively acquire the heat map, depth map and RGB image. The depth map acquired in the embodiment of the present application is point cloud data, and the point cloud data acquired in step S201 and the point cloud data acquired by the depth map of this step can be acquired by the same image acquisition device.

[0121] S210, converting the heat map and the RGB image into the coordinate system of the depth map, and inputting the converted data and the depth map into a predetermined deep learning network model to obtain joint prediction data.

[0122] In this embodiment, the deep learning network model can be specifically understood as a pre-trained neural network model for identifying human joint data; the joint prediction data can be specifically understood as data such as the coordinates of human joints obtained by model prediction.

[0123] The deep learning network model is pre-trained, and the joint types and joint numbers that can be predicted by the deep learning network model can also be set according to the needs. It is also possible to use a deep learning network model that has determined the joint types and joint numbers, or share the deep learning network model with other scenes. Since the position of each camera in the same image acquisition device is determined, its coordinate system is also determined, so the coordinate conversion relationship between the cameras is also determined. According to the coordinate conversion relationship between the cameras, the heat map and the RGB image are respectively converted to the coordinate system where the depth map is located. In addition, in the case where multiple image acquisition devices collect data, the present application can also perform matching and alignment of point cloud data. The matching and alignment method can use the ICP algorithm to uniformly convert all data to the coordinate system where an image acquisition device is located. The converted heat map, the converted RGB image and the depth map are simultaneously input into the pre-trained deep learning network model to obtain the joint prediction data predicted and output by the model.

[0124] S211, convert the joint prediction data into the base coordinate system to obtain candidate prediction data.

[0125] In this embodiment, the candidate prediction data can be specifically understood as coordinate data under the base coordinates.

[0126] The joint prediction data obtained in the embodiment of the present application is data in the coordinate system corresponding to the depth map, which is converted to the base coordinate system of the robot arm through the coordinate conversion relationship from the camera to the base to obtain candidate prediction data. The conversion method of this step is the same as the conversion method of S202, and will not be repeated here.

[0127] S212: Filter unnecessary joints of each candidate prediction data to determine target prediction data.

[0128] In this embodiment, the target prediction data can be specifically understood as joint coordinate data used to determine human body obstacles.

[0129] Some joints of the human body will not affect the normal operation of the robot arm. Therefore, after predicting the joint data, the joint data can be filtered to remove unnecessary joints. The unnecessary joints in this step can be used for other functions. The embodiment of the present application can pre-set the joints that need to be filtered out. After determining the candidate prediction data, the coordinate data corresponding to each joint can be determined accordingly. Non-essential joints are filtered according to the pre-set joints that need to be filtered out to obtain target prediction data.

[0130] Usually, the robot arm has a base, and the human joint points under the base do not affect the work of the robot arm, so there is no need to consider the joint nodes of the lower body. Therefore, when filtering non-essential joints, the present application can filter the joints of the lower body, as well as non-essential joint nodes such as eyes and fingers.

[0131] The embodiment of the present application can also train the deep learning network model in a targeted manner so that all joint nodes predicted by it are necessary joint nodes. That is, after the joint prediction data predicted by the deep learning network model is converted to the base coordinate system, the candidate prediction data is directly used as the target prediction data without the need for unnecessary joint filtering.

[0132] Exemplarily, an embodiment of the present application provides a method for determining target prediction data, which inputs a heat map, a depth map, and an RGB image, and outputs the 3D coordinates of 34 joint nodes of the human body. For occluded joint nodes, their coordinates are determined by inference by the model, and according to the credibility of the inference, there are four confidence levels from high to low, namely: high (HIGH), medium (MEDIUM), low (LOW), and joints are occluded (NONE). The joint points acquired by multiple cameras are weighted according to their own confidence levels. The I-th joint node m acquired by camera j ji The coordinates of (X ij , Y ij , Z ij ), whose confidence level is α j , the i-th joint node m acquired by camera k ki The coordinates of (Xik , Y ik , Z ik ), the confidence level is α k , if α j =α k , then take the average value of the joint node coordinates as the result, If α j >α k The joint node coordinates obtained by the camera with greater confidence are taken as the result, m i =max(m ki , m ji ). For human-computer interaction scenarios, if a bounding sphere is used to represent the human body, the 34 joint nodes obtained, such as eyes and fingers, are unnecessary joint nodes. After preprocessing the joint nodes, redundant unnecessary nodes are deleted, and only 18 necessary nodes are retained.

[0133] The embodiments of the present application can improve the confidence level of human joint nodes under occlusion conditions, and obtain high-precision human joint coordinates, namely target prediction data.

[0134] S213. Represent the predicted data of each target by a surrounding sphere to obtain obstacle information of the obstacle.

[0135] Each target prediction data is represented by an enclosing sphere. In this step, each joint is represented by an enclosing sphere, that is, each target prediction data corresponding to each joint is represented by an enclosing sphere to obtain obstacle information of the obstacle. Different joints can be represented by enclosing spheres of different sizes. The size of the enclosing sphere depends on the type of joint. For larger joints, a larger enclosing sphere can be set, and for smaller joints, a smaller enclosing sphere can be set. For each joint, the size of its enclosing sphere can be determined according to the average length and width of the human body at that part. For example, the average length and width of the human body is used as the radius of the enclosing sphere.

[0136] S214: Determine obstacle information of the static obstacle according to the pre-built static obstacle model.

[0137] In this embodiment, the static obstacle model can be specifically understood as a model describing a static obstacle, which can be used to describe information such as the size, position, and shape of the static obstacle; the static obstacle can be an object that remains stationary, such as a workbench. The predicted position information can be information indicating the position of the static obstacle, such as the center point coordinates, length, width, and outer contour coordinates.

[0138] A static obstacle model is constructed in advance according to information such as the size, position, and shape of static obstacles. When constructing the static obstacle model, it can be directly constructed in the base coordinate system to reduce subsequent coordinate conversion work. If other objects are used as references for static obstacles, their coordinates need to be converted to the base coordinate system after determining their coordinates. The static obstacle model in the embodiments of the present application can be constructed offline. After the static obstacle model is constructed, if the static obstacle does not change, it can be used all the time without detecting the static obstacle every time when performing obstacle detection. The obstacle information of the static obstacle is determined according to the information such as the position, coordinates, and shape of the obstacle defined in the static obstacle model.

[0139] It should be noted that S201 - S208 in the embodiments of the present application is a method for determining obstacle information and can identify any type of obstacle in space; S209 - S213 is a method for determining obstacle information and identifies the human body in space; S214 is a method for determining obstacle information and determines static obstacles. The above several methods are parallel steps and there is no strict sequence in execution.

[0140] S215. Determine the obstacle distance between the obstacle and the robotic arm according to each obstacle information.

[0141] In the embodiments of the present application, the distance calculation method can be determined in advance, for example, the GJK algorithm. Since the obstacle representations are all convex hulls such as bounding spheres, bounding boxes, or OCTOmaps, and the URDF model of the robotic arm is also a convex hull, the difference sets in different directions are iterated to find the closest support points of the two polygons as the basis for calculating the minimum distance, and the minimum distance between the two convex hulls is calculated.

[0142] S216. Determine the obstacle information in the safety zone, the obstacle information in the monitoring zone, and the obstacle information in the cooperation zone according to each obstacle distance in combination with the obstacle classification strategy.

[0143] In this embodiment, the safety zone, the monitoring zone, and the cooperation zone can also be divided by distance. For example, when the obstacle distance d < 1m, it is the cooperation zone; when 1m < d < 3m, it is the monitoring zone; when d > 3m, it is the safety zone.

[0144] The obstacle classification strategy is determined in advance. After determining the distances of each obstacle, the obstacle belongs to which zone is determined in combination with the obstacle classification strategy, and then the obstacle information in the safety zone, the obstacle information in the monitoring zone, and the obstacle information in the cooperation zone are obtained.

[0145] S217. Determine the number of joint nodes in the safety zone according to the obstacle information in the safety zone, and determine the number of human bodies according to the number of joint nodes. The number of human bodies is used to remind the staff.

[0146] For the safety zone, human activities will not affect the normal operation of the robot arm. Therefore, the number of joint nodes in the safety zone is identified, and the number of human bodies is determined based on the number of joint nodes. When identifying the number of joint nodes, the embodiment of the present application can only identify one type of joint node number, for example, only the number of heads is identified, and the number of heads can represent the number of human bodies. The staff is prompted based on the number of human bodies, and the prompting method can be to display text on the operation screen, or to play voice, etc.

[0147] S218. Determine human body information in the monitoring area according to the obstacle information in the monitoring area, represent the human body information by an enclosing sphere, determine the obstacle distance corresponding to the human body information and display it.

[0148] For the monitoring area, the speed of objects in the monitoring area will not change suddenly, so only human obstacles can be detected. Determine the human information in the monitoring area, which can be the position of the human joints such as the head, shoulders, wrists, and feet. Attach the enclosing sphere to the human body and display the enclosing sphere corresponding to the human information; further determine the obstacle distance corresponding to the human information, that is, the distance between the human body and the robotic arm, and display this distance to the staff. The obstacle distance corresponding to the human information can also be broadcasted by voice to remind the staff.

[0149] S219: Represent obstacle information in the collaboration area through a bounding box.

[0150] For the collaboration area, the safety factor of this area is low. In order to ensure that the robotic arm can avoid obstacles in time, a bounding box is attached to the obstacle information in the collaboration area and displayed to the staff.

[0151] For example, Figure 4 An example diagram of an obstacle detection process is provided, including dynamic obstacle detection and static obstacle modeling, wherein the dynamic obstacle detection includes human body detection and object detection.

[0152] S1. Calibrate multiple cameras.

[0153] S2. Point cloud matching is performed using the ICP algorithm.

[0154] The point cloud matching in the embodiment of the present application may be matching the point cloud data in the base coordinate system.

[0155] When representing the human body, it is achieved through steps S3-S5.

[0156] S3. Obtain joint prediction data through deep learning network model prediction.

[0157] In this step, the joint coordinates can be predicted through the deep learning network model to obtain joint prediction data.

[0158] S4. Non-essential joint filtering.

[0159] This step simplifies and fuses unnecessary joints, filters out unnecessary joints, and obtains target prediction data.

[0160] S5. Add a bounding ball to the joint.

[0161] When representing an object, it is achieved through steps S6-S7.

[0162] S6. Point cloud segmentation and clustering.

[0163] This step can segment and cluster the target point cloud data to obtain clustered point cloud data.

[0164] S7. Bounding box representation.

[0165] In this step, the clustered point cloud data can be represented by a bounding box.

[0166] When determining the static obstacle, it is achieved through steps S8-S9.

[0167] S8, point cloud filtering removes environmental information.

[0168] Filter the point cloud data in the environment to filter out the environmental information of non-static obstacles.

[0169] S9. Offline modeling of static obstacle model.

[0170] S10. Calculate the distance between the obstacle and the robotic arm to obtain the obstacle distance.

[0171] S11. Gradual representation based on obstacle distance.

[0172] In this step, obstacles of different distances can be graded by using an obstacle classification strategy.

[0173] The embodiment of the present invention provides an obstacle detection method, which solves the problems of inaccurate obstacle detection and high detection cost, effectively improves the accuracy of obstacle detection, reduces the amount of data processing, improves data processing efficiency, and realizes multi-level perception of obstacles by performing obstacle recognition through various methods such as obstacle detection, static obstacle modeling, and human joint recognition. The embodiment of the present application replaces static obstacle point cloud recognition with offline modeling, which simplifies the obstacle recognition and data processing process; and simplifies dynamic obstacle and human obstacle data through noise reduction, downsampling, etc. in the process of dynamic obstacle recognition; and uses the bounding box method OBB with low computational complexity to represent obstacles, realizing high-speed recognition. And through the obstacle classification strategy, hierarchical perception of the environment is realized.

[0174] Embodiment 3

[0175] Figure 5 A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0176] like Figure 5 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0177] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0178] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs the various methods and processes described above, such as the obstacle detection method.

[0179] In some embodiments, the obstacle detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the obstacle detection method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the obstacle detection method in any other appropriate manner (e.g., by means of firmware).

[0180] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0182] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0183] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0184] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0185] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0186] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0187] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An obstacle detection method, characterized in that: include: Acquire point cloud data collected by at least two image acquisition devices, and determine candidate point cloud data according to each of the point cloud data combined with a corresponding camera-to-base coordinate transformation relationship, wherein each of the candidate point cloud data is in the same coordinate system; Filtering each of the candidate point cloud data to obtain target point cloud data; Perform obstacle identification according to each of the target point cloud data to determine obstacle information; The obstacle distance between the obstacle and the robot arm is determined according to the obstacle information, and the obstacles are graded and represented according to the obstacle distances combined with the obstacle classification strategy.

2. The method according to claim 1, characterized in that The step of determining candidate point cloud data according to each of the point cloud data in combination with a corresponding camera-to-base coordinate transformation relationship includes: For the point cloud data collected by each image acquisition device, each point cloud data is converted to the base coordinate system through the corresponding camera to base coordinate conversion relationship to obtain the corresponding base point cloud data; If the image acquisition device is a reference device, each of the base point cloud data is used as candidate base point cloud data; If the image acquisition device is a device to be converted, the point cloud data of each base is used as the point cloud data to be processed; Determine a coordinate conversion matrix based on each of the to-be-processed point cloud data in combination with the base point cloud data corresponding to the image acquisition device as a reference device, and convert each of the to-be-processed point cloud data into candidate base point cloud data based on the coordinate conversion matrix; Deduplication processing is performed on each of the candidate base point cloud data to obtain candidate point cloud data.

3. The method according to claim 2, characterized in that The step of determining the coordinate transformation relationship from the camera to the base includes: Determine the posture transformation relationship of the image acquisition device as a reference device under two robot arm posture conditions, and determine a first coordinate transformation matrix between the reference device and the robot arm base according to the posture transformation relationship, wherein the first coordinate transformation matrix is ​​a coordinate transformation relationship between the camera corresponding to the image acquisition device as the reference device and the base; For each image acquisition device as a device to be converted, a second coordinate transformation matrix from the first calibration plate to the base of the manipulator is determined in combination with the first coordinate transformation matrix according to the coordinate transformation relationship from the first calibration plate to the reference device, wherein the first calibration plate is fixed in the base coordinate system of the manipulator; According to the second coordinate transformation matrix combined with the coordinate transformation relationship between the device to be transformed and the first calibration plate, the third coordinate transformation matrix from the device to be transformed to the base of the robotic arm is determined. The third coordinate transformation matrix is ​​the coordinate transformation relationship from the camera corresponding to the image acquisition device of the device to be transformed to the base.

4. The method according to claim 1, characterized in that: The filtering of each candidate point cloud data to obtain target point cloud data includes: Downsampling and denoising the candidate point cloud data to obtain denoised point cloud data; Fitting a plane according to each of the noise-reduced point cloud data, and determining the largest fitted plane; The noise-reduced point cloud data in the maximum plane is filtered to obtain target point cloud data.

5. The method according to claim 1, characterized in that The step of identifying obstacles according to each of the target point cloud data and determining obstacle information includes: Clustering each of the target point cloud data to obtain clustered point cloud data; Each of the clustered point cloud data is represented by a bounding box to obtain obstacle information of the obstacle.

6. The method according to claim 1, characterized in that Before determining the obstacle distance between the obstacle and the robotic arm according to each obstacle information, the method further includes: Acquire the thermal map, depth map and RGB image collected by each of the image acquisition devices, wherein the depth map is point cloud data; Converting the thermal map and the RGB image to the coordinate system of the depth map, and inputting the converted data and the depth map into a predetermined deep learning network model to obtain joint prediction data; Convert the joint prediction data into a base coordinate system to obtain candidate prediction data; Performing non-essential joint filtering on each of the candidate prediction data to determine target prediction data; The target prediction data are represented by a surrounding sphere to obtain obstacle information of the obstacle.

7. The method according to claim 1, characterized in that Before determining the obstacle distance between the obstacle and the robotic arm according to each obstacle information, the method further includes: Obstacle information of the static obstacle is determined according to a pre-built static obstacle model.

8. The method according to any one of claims 1 to 7, characterized in that: The step of performing classification of the obstacles according to the distance of each obstacle in combination with the obstacle classification strategy includes: Determine obstacle information of the safety zone, obstacle information of the monitoring zone, and obstacle information of the cooperation zone according to the obstacle distances and the obstacle classification strategy; Determine the number of joint nodes in the safety zone according to the obstacle information in the safety zone, determine the number of human bodies according to the number of joint nodes, and the number of human bodies is used to remind the staff; Determine human body information in the monitoring area according to obstacle information in the monitoring area, represent the human body information by a surrounding sphere, determine the obstacle distance corresponding to the human body information and display it; The obstacle information in the collaboration area is represented by a bounding box.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the obstacle detection method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the obstacle detection method according to any one of claims 1 to 8 when executed.

Citation Information

Patent Citations

  • Obstacle detection method, vehicle, equipment and computer storage medium

    CN113536883A

  • Obstacle detection method and device, electronic equipment and storage medium

    CN114091515A

  • Automatic obstacle avoidance method and device for mechanical arm

    CN115056225A

  • Obstacle detection method and device, electronic equipment and storage medium

    CN115685249A

  • Method and system for sensing automated driving environment

    WO2022022694A1