Robot collision warning method, device, equipment, medium and program product

Abstract

The application provides a robot collision warning method, device, equipment, medium and program product. In the method, the point cloud data of each robot and the robot identifier, and the point cloud data of each obstacle are determined according to the acquired robot working scene image shot by a camera device, substation point cloud data and internal and external parameters of the camera device. Then, the moving direction of the robot is determined according to the acquired robot historical point cloud data, the point cloud data and the robot identifier of the robot. The distance and the relative direction between the robot and the obstacle are calculated according to the point cloud data of the robot and the point cloud data of the obstacle. Finally, collision warning is performed according to the moving direction of the robot, the distance and the relative direction between the robot and the obstacle. The robot working scene image, the substation point cloud data, the robot historical point cloud data and the internal and external parameters of the camera device are used to realize collision warning when the sensor of the robot is damaged.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a robot collision warning method, device, equipment, medium and program product. BACKGROUND

[0002] In order to improve the safety of the staff and reduce the workload of the staff, in the substation, robots can be used for inspection, operation of equipment and other work.

[0003] In the prior art, sensors such as laser radar, ultrasonic sensor, millimeter wave radar are installed in the robot, which can detect whether there is an obstacle around the robot, and then determine whether a collision will occur. When it is determined that a collision will occur, a warning is given.

[0004] However, when the sensor installed in the robot is damaged, collision warning cannot be achieved, so there is an urgent need for a robot collision warning method that can achieve collision warning when the sensor installed in the robot is damaged. SUMMARY

[0005] The robot collision warning method, device, equipment, medium and program product provided by the embodiments of the present application solve the problem that in the prior art, when the sensor installed in the robot is damaged, collision warning cannot be achieved.

[0006] In a first aspect, the embodiments of the present application provide a robot collision warning method, comprising:

[0007] According to the robot working scene image captured by the camera device, the substation point cloud data and the internal and external parameters of the camera device, the point cloud data and the robot identifier of each robot in the robot working scene image, and the point cloud data of each obstacle are determined;

[0008] For each robot, according to the obtained robot historical point cloud data, the point cloud data and the robot identifier of the robot, the moving direction of the robot is determined;

[0009] According to the point cloud data of each robot and the point cloud data of each obstacle, the distance and the relative direction between each robot and each obstacle are calculated;

[0010] According to the moving direction of each robot, the distance and the relative direction between each robot and each obstacle, collision warning is performed.

[0011] In a possible implementation, the determination of the point cloud data and the robot identifier of each robot in the robot working scene image according to the robot working scene image captured by the camera device, the substation point cloud data and the internal and external parameters of the camera device comprises:

[0012] inputting the robot working scene image, the substation point cloud data and the internal and external parameters of the camera equipment into a first mapping model to obtain point cloud data corresponding to each pixel point in the robot working scene image; the first mapping model is a deep learning model that is pre-trained to determine point cloud data corresponding to each pixel point in an image according to the image, the point cloud data and the internal and external parameters of the camera equipment;

[0013] performing target detection on the robot working scene image to obtain robot data of each robot and image positions of each obstacle, the robot data including the image positions and robot identifiers;

[0014] determining point cloud data of each robot and point cloud data of each obstacle according to the point cloud data corresponding to each pixel point in the robot working scene image, the image positions of each robot and the image positions of each obstacle.

[0015] In a possible implementation, the collision warning according to the moving direction of each robot, the distance and relative direction between each robot and each obstacle includes:

[0016] For each robot and each obstacle, if the distance between the robot and the obstacle is less than a preset distance and the included angle between the relative direction of the robot and the obstacle and the moving direction of the robot belongs to a preset angle range, a collision warning is performed.

[0017] In a possible implementation, the robot data of each robot further includes a pose, and before determining the moving direction of each robot according to the obtained robot historical point cloud data, the point cloud data of the robot and the robot identifier, the method further includes:

[0018] For each robot, determining a contact ground position of the robot according to the image position of the robot;

[0019] For each robot, inputting the contact ground position of the robot, the pose of the robot and the substation point cloud data into a second mapping model to obtain updated point cloud data of the robot; the second mapping model is a deep learning model that is pre-trained to determine point cloud data corresponding to a robot according to the contact ground position, the pose and the point cloud data of the robot.

[0020] In a possible implementation, the determination of the moving direction of each robot according to the obtained robot historical point cloud data, the point cloud data of the robot and the robot identifier includes:

[0021] According to the acquired robot historical point cloud data, the updated point cloud data of the robot and the robot identifier, a moving direction of the robot is determined.

[0022] The distance and relative direction of each robot to each obstacle are calculated according to the point cloud data of each robot and the point cloud data of each obstacle.

[0023] The distance and relative direction of each robot to each obstacle are calculated according to the updated point cloud data of each robot and the point cloud data of each obstacle.

[0024] In a possible implementation, the method further comprises:

[0025] Acquiring user manipulation data, the user manipulation data comprising a clicked robot image position and a clicked target site image position;

[0026] According to the point cloud data corresponding to each pixel point in the robot working scene image, the clicked robot image position and the clicked target site image position, a clicked robot point cloud data and a target site are determined.

[0027] According to the clicked robot point cloud data, the point cloud data of each robot and the robot identifier of each robot, a target robot identifier is determined.

[0028] The robot corresponding to the target robot identifier is controlled to move to the target site.

[0029] In a possible implementation, if the user manipulation data further comprises a clicked article image position, the method further comprises:

[0030] According to the point cloud data corresponding to each pixel point in the robot working scene image and the clicked article image position, an article point cloud position is determined.

[0031] According to the article point cloud position and the target site, a target orientation is determined.

[0032] The orientation of the robot corresponding to the target robot identifier is controlled to be the target orientation.

[0033] In a second aspect, the embodiments of the present application provide a robot collision warning device, comprising:

[0034] The processing module is configured to:

[0035] According to the acquired robot working scene image shot by the camera device, the substation point cloud data and the internal and external parameters of the camera device, the point cloud data of each robot and the robot identifier, and the point cloud data of each obstacle are determined.

[0036] For each robot, according to the obtained robot historical point cloud data, the point cloud data of the robot and the robot identifier, a moving direction of the robot is determined;

[0037] According to the point cloud data of each robot and the point cloud data of each obstacle, a distance and a relative direction of each robot to each obstacle are calculated;

[0038] The warning module is configured to perform collision warning according to the moving direction of each robot, the distance and the relative direction of each robot to each obstacle.

[0039] In a third aspect, an electronic device is provided, and the electronic device comprises:

[0040] a processor, a memory and a communication interface;

[0041] The memory is configured to store executable instructions of the processor;

[0042] The processor is configured to perform the robot collision warning method according to any one of the first aspect by executing the executable instructions.

[0043] In a fourth aspect, a readable storage medium is provided, and the readable storage medium stores a computer program, and the computer program is configured to implement the robot collision warning method according to any one of the first aspect when executed by a processor.

[0044] In a fifth aspect, a computer program product is provided, and the computer program product comprises a computer program, and the computer program is configured to implement the robot collision warning method according to any one of the first aspect when executed by a processor.

[0045] The robot collision warning method, device, equipment, medium and program product provided by the embodiments of the present application determine the point cloud data of each robot and the robot identifier in the robot working scene image, and the point cloud data of each obstacle, according to the obtained robot working scene image, substation point cloud data and internal and external parameters of the camera device; then, the moving direction of the robot is determined according to the obtained robot historical point cloud data, the point cloud data of the robot and the robot identifier; the distance and the relative direction of each robot to each obstacle are calculated according to the point cloud data of each robot and the point cloud data of each obstacle; finally, collision warning is performed according to the moving direction of each robot, the distance and the relative direction of each robot to each obstacle. The present application realizes collision warning when the sensor of the robot is damaged, by using the robot working scene image, the substation point cloud data, the robot historical point cloud data and the internal and external parameters of the camera device. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0047] Figure 1 A flowchart of a robot collision warning method embodiment provided by the present application;

[0048] Figure 2 A flowchart of a robot collision warning method embodiment provided by the present application;

[0049] Figure 3 A flowchart of a robot collision warning method embodiment provided by the present application;

[0050] Figure 4 A structural diagram of a robot collision warning device embodiment provided by the present application;

[0051] Figure 5 A structural diagram of an electronic device provided by the present application.

[0052] The above drawings summarize the embodiments of the present application, which will be described in more detail hereinafter. These drawings and the written description are not intended to constrain the scope of the inventive concept by any means, but to illustrate the inventive concept to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0053] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements, unless the context clearly dictates otherwise. The following exemplary embodiments are described in enough detail to enable those skilled in the art to make and use the application. The embodiments described herein are not meant to limit the application to a particular embodiment, but rather the scope of the application is to be determined with reference to the appended claims along with the full description of the application presented herein.

[0054] The terms "first", "second", "third", "fourth" and the like in the description and in the claims, if any, of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms is interchangeable under appropriate circumstances such that the embodiments of the present application described herein are capable of accomplishing functionalities that are either the same or similar to other embodiments of the present application described herein, without commencing to violate the principles of the present application. Moreover, the terms "include", "have", and the like, as well as any variations thereof, are intended to cover a non-exclusive inclusion such that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements but can include other elements not expressly listed or inherent to such process, method, system, product, or apparatus.

[0055] With the development of science and technology, dangerous or repetitive work in the field of power grid can be performed by robots, such as inspection and operation of equipment in a substation by using robots.

[0056] In the prior art, sensors such as laser radar, ultrasonic sensor, millimeter wave radar are installed in the robot, and whether there is an obstacle around the robot during movement can be detected, and whether a collision will occur is determined. When it is determined that a collision will occur, a warning is given.

[0057] However, when the sensors installed in the robot are damaged, collision warning cannot be achieved, so there is an urgent need for a robot collision warning method that can achieve collision warning when the sensors installed in the robot are damaged.

[0058] In view of the problems in the prior art, the inventors found, in the process of researching the robot collision warning method, that in order to achieve collision warning when the sensors installed in the robot are damaged, a camera device can be installed in the substation, the camera device can capture a robot working scene image, and the image includes the robot. A laser radar is installed in the substation to collect substation point cloud data. The internal and external parameters of the robot working scene image, the substation point cloud data and the camera device are used to determine the point cloud data of each robot and the robot identifier in the robot working scene image, and the point cloud data of each obstacle. Then, in combination with the historical point cloud data of the robot, the moving direction of the robot,

[0059] The distance and relative direction of each robot and each obstacle are determined, and collision warning is performed without the data of the sensors of the robot. Based on the above invention concept, the robot collision warning scheme in the present application is designed.

[0060] The execution subject of the robot collision warning method in the present application can be a computer, and can also be a server, a terminal device, etc., which is not limited in the present application, and the computer will be taken as an example for illustration.

[0061] The application scenario of the robot collision warning method provided in the present application will be described by way of example.

[0062] For example, in this application scenario, a camera device and a laser radar are installed in the substation, the camera device can capture a robot working scene image and send it to the computer, and the laser radar can collect substation point cloud data and send it to the computer. The robot working scene image includes the robot, and the data of each point in the substation point cloud data includes three-dimensional coordinates.

[0063] The robot inspects in the substation, detects that its own sensor is damaged, and cannot perform collision warning, and sends a collision warning instruction to the computer.

[0064] After the computer receives the collision warning instruction, the computer determines, according to the robot working scene image, the substation point cloud data, and the internal and external parameters of the camera equipment, the point cloud data and the robot identifier of each robot in the robot working scene image, and the point cloud data of each obstacle.

[0065] Further, for each robot, the moving direction of the robot is determined according to the obtained robot historical point cloud data, the point cloud data and the robot identifier of the robot. The distance and the relative direction between each robot and each obstacle are calculated according to the point cloud data of each robot and the point cloud data of each obstacle.

[0066] Further, the collision warning is performed according to the moving direction of each robot, the distance and the relative direction between each robot and each obstacle. When it is determined that a robot is about to collide, a collision warning message can be sent to the robot and the terminal device of the worker, and the robot can stop moving to avoid collision.

[0067] It should be noted that the above scenario is only an example of an application scenario provided by the embodiments of the present application, and the actual form of various devices included in the scenario is not limited, nor is the interaction mode between the devices limited. In the specific application of the scheme, the actual needs can be set according to the actual needs.

[0068] In the following, the technical scheme of the present application is described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments.

[0069] Figure 1 The flowchart of the robot collision warning method embodiment provided by the present application is shown in the present embodiment. The present embodiment describes the collision warning performed by the computer according to the robot working scene image, the substation point cloud data, the robot historical point cloud data, and the internal and external parameters of the camera equipment. The method in the present embodiment can be realized by software, hardware, or a combination of software and hardware. As shown in the figure, the robot collision warning method specifically includes the following steps: Figure 1

[0070] S101: According to the robot working scene image, the substation point cloud data, and the internal and external parameters of the camera equipment, the point cloud data and the robot identifier of each robot in the robot working scene image, and the point cloud data of each obstacle are determined.

[0071] ​The camera device and the laser radar are installed in the transformer substation. The camera device captures the robot working scene image and sends it to the computer, and at the same time, the internal and external parameters of the camera device are sent to the computer. The laser radar collects the point cloud data of the transformer substation and sends it to the computer. The time of collecting data by the camera device and the laser radar is the same. The robot working scene image includes the robot, and the data of each point in the point cloud data of the transformer substation includes the three-dimensional coordinates in the world coordinate system.

[0072] It should be noted that if the robot is installed with a camera, the image captured by the camera can also be used as the robot working scene image.

[0073] In this step, in order to realize collision warning when the sensor installed on the robot is damaged, the computer first determines the point cloud data of each robot and the identification of the robot in the robot working scene image, and the point cloud data of each obstacle according to the robot working scene image captured by the camera device, the point cloud data of the transformer substation and the internal and external parameters of the camera device.

[0074] Specifically, the robot working scene image, the point cloud data of the transformer substation and the internal and external parameters of the camera device are input into the first mapping model to obtain the point cloud data corresponding to each pixel point in the robot working scene image. The first mapping model is a pre-trained deep learning model for determining the point cloud data corresponding to each pixel point in the image according to the image, the point cloud data and the internal and external parameters of the camera device.

[0075] The external parameters of the camera device include a translation matrix and a rotation matrix. The translation matrix is used to describe the translation relationship of the camera coordinate system relative to the world coordinate system, which is composed of three translation components in x, y and z directions, and can determine the position of the camera in the world coordinate system. The rotation matrix is used to describe the rotation relationship of the camera coordinate system relative to the world coordinate system, which is composed of rotation angles around the x, y and z axes of the camera coordinate system, and can represent the orientation and attitude of the camera.

[0076] The internal parameters of the camera device include focal length, principal point, distortion parameter, image size and scaling factor.

[0077] The focal length is the focal length of the camera, usually in units of pixels, which is used to describe the convergence ability of the lens to light, and determines the angle of view of the image and the size ratio of the object in the image.

[0078] The principal point refers to the intersection of the optical axis and the image plane, which is in units of pixels.

[0079] The distortion parameter is used to describe the shape distortion of the lens, including radial distortion and tangential distortion. Radial distortion causes the deviation of image center and edge pixel positions, such as barrel distortion or pillow distortion; tangential distortion is caused by the non-parallel or non-perpendicular of the lens and the image plane.

[0080] Image size is used to describe the width and height of the image, which determines the resolution and the number of pixels of the image, and affects the ability to capture the details of the object.

[0081] Scaling factor is used to describe the scaling difference in x and y directions.

[0082] Target detection is performed on the robot working scene image to obtain robot data of each robot and image position of each obstacle, and the robot data includes the image position and the robot identifier.

[0083] According to the point cloud data corresponding to each pixel point in the robot working scene image, the image position of each robot and the image position of each obstacle, the point cloud data of each robot and the point cloud data of each obstacle are determined.

[0084] For each robot, the image position of the robot is determined, and the point cloud data corresponding to the pixel points included in the image position is determined, thereby obtaining the point cloud data of the robot.

[0085] For each obstacle, the image position of the obstacle is determined, and the point cloud data corresponding to the pixel points included in the image position is determined, thereby obtaining the point cloud data of the obstacle.

[0086] The point cloud data of each robot includes data of at least one point, and the point cloud data of each obstacle includes data of at least one point.

[0087] It should be noted that the condition for triggering the computer to perform robot collision warning can be: receiving the collision warning instruction sent by the robot, receiving the collision warning instruction sent by the staff through the terminal device, and reaching the preset collision warning time. The embodiment of the present application does not limit the condition for triggering the computer to perform robot collision warning, which can be determined according to the actual situation.

[0088] S102: For each robot, according to the obtained robot historical point cloud data, the point cloud data of the robot and the robot identifier, the moving direction of the robot is determined.

[0089] In this step, after the computer determines the point cloud data of each robot and the robot identifier, in order to determine whether the robot is about to collide, for each robot, according to the obtained robot historical point cloud data, the point cloud data of the robot and the robot identifier, the moving direction of the robot is determined.

[0090] Specifically, the robot historical point cloud data includes a plurality of correspondence data, each correspondence data includes a robot identifier and its corresponding historical point cloud data. It is first determined whether the robot historical point cloud data includes the robot identifier of the robot.

[0091] If the robot identifier of the robot is included in the robot historical point cloud data, the historical point cloud data corresponding to the robot identifier of the robot is obtained from the robot historical point cloud data. The average value of all three-dimensional coordinates in the historical point cloud data is taken as a first coordinate, the average value of all three-dimensional coordinates in the point cloud data of the robot is taken as a second coordinate, and the direction from the second coordinate to the first coordinate is taken as the moving direction of the robot.

[0092] If the robot identifier of the robot is not included in the robot historical point cloud data, a collision warning failure prompt message is output to remind the staff to check whether the robot will collide in time.

[0093] It should be noted that the computer can obtain the robot historical point cloud data in the following ways: the staff uses a terminal device to send the robot historical point cloud data to the computer, and the computer can obtain the robot historical point cloud data. It can also be that the camera device and the laser radar synchronously collect data and send the data to the computer, and the computer can perform collision warning after receiving the robot working scene image and the substation point cloud data each time, so that the point cloud data and the robot identifier of each robot obtained in the last collision warning process of the current collision warning are taken as the historical point cloud data. It can also be that the staff inputs the robot historical point cloud data into the computer, and the computer can obtain the robot historical point cloud data. The embodiments of the present application do not limit the way in which the computer obtains the robot historical point cloud data, and the way can be determined according to the actual situation.

[0094] S103: According to the point cloud data of each robot and the point cloud data of each obstacle, the distance and the relative direction between each robot and each obstacle are calculated.

[0095] In this step, after the computer obtains the point cloud data of each robot and the point cloud data of each obstacle, the distance and the relative direction between each robot and each obstacle are calculated according to the point cloud data of each robot and the point cloud data of each obstacle.

[0096] For each robot and each obstacle, the following processing is performed:

[0097] The average value of all three-dimensional coordinates in the point cloud data of the robot is taken as a third coordinate, and the average value of all three-dimensional coordinates in the point cloud data of the obstacle is taken as a fourth coordinate.

[0098] The direction from the third coordinate to the fourth coordinate is taken as the relative direction between the robot and the obstacle. The distance between the position corresponding to the third coordinate and the position corresponding to the fourth coordinate is taken as the distance between the robot and the obstacle.

[0099] It should be noted that the execution sequence of step S102 and step S103 can be: first executing step S102, and then executing step S103. It can also be: first executing step S103, and then executing step S102. It can also be: step S102 and step S103 are executed at the same time. The execution sequence of step S102 and step S103 is not limited in the embodiment of the present application, and can be determined according to actual conditions.

[0100] S104: Collision warning is performed according to the moving direction of each robot, the distance and relative direction between each robot and each obstacle.

[0101] In this step, after the computer obtains the moving direction of each robot, the distance and relative direction between each robot and each obstacle, collision warning can be performed.

[0102] Specifically, for each robot and each obstacle, if the distance between the robot and the obstacle is less than a preset distance, and the included angle between the relative direction of the robot and the obstacle and the moving direction of the robot belongs to a preset angle range, it indicates that the robot moves towards the obstacle and the distance is close, that is, a collision will occur, and then collision warning is performed.

[0103] It should be noted that the preset distance can be 0.3 meters, 0.5 meters, 1 meter, etc., and the preset angle range can be 0-5 degrees, 0-10 degrees, 0-30 degrees, etc. The preset distance and the preset angle range are not limited in the embodiment of the present application, and can be determined according to actual conditions.

[0104] It should be noted that for each robot, the computer can send the point cloud data of the robot and the point cloud data of each obstacle to the robot, so that the robot can avoid obstacles.

[0105] It should be noted that the computer can also calculate the average value of the three-dimensional coordinates in the point cloud data of all robots as the rotation target coordinate. Then, according to the corresponding relationship between the coordinates and the angles, the rotation target angle corresponding to the rotation target coordinate is determined, and the rotation target angle is sent to the camera device. The camera device rotates to the rotation target angle, so that the robot can be in the center part of the shooting area of the camera device.

[0106] The robot collision warning method provided in the embodiment comprises the following steps.

[0107] Figure 2 The flowchart of the second embodiment of the robot collision warning method provided in the application is shown in the above embodiment. The application embodiment describes the situation of updating the point cloud data of the robot. As shown in the figure, Figure 2

[0108] S201: For each robot, determine the contact ground position of the robot according to the image position of the robot.

[0109] In this step, in order to improve the accuracy of collision warning, the point cloud data of the robot can be updated. After determining the point cloud data of each robot and the robot identifier, and the point cloud data of each obstacle, the computer determines the contact ground position of each robot according to the image position of the robot.

[0110] Specifically, the pixel points corresponding to the image position of the robot in the robot working scene image are determined, and then the positions of the pixel points in the lowest preset number of rows of the pixel points are taken as the contact ground position of the robot.

[0111] It should be noted that the preset number can be 1, 2, 3, etc. The application embodiment does not limit the preset number, which can be determined according to the actual situation.

[0112] S202: For each robot, input the contact ground position of the robot, the attitude of the robot and the substation point cloud data into the second mapping model to obtain the updated point cloud data of the robot.

[0113] ​In this step, after the computer obtains the contact ground position of each robot, because the robot data of each robot also includes the pose, for each robot, the contact ground position of the robot, the pose of the robot and the substation point cloud data are input into the second mapping model to obtain the updated point cloud data of the robot.

[0114] The second mapping model is a pre-trained deep learning model for determining the point cloud data corresponding to the robot according to the contact ground position, the pose and the point cloud data of the robot.

[0115] It should be noted that the pose can be standing, squatting, lying and the like, and the embodiments of the present application do not limit the pose, which can be determined according to actual conditions.

[0116] S203: For each robot, the moving direction of the robot is determined according to the obtained robot historical point cloud data, the updated point cloud data of the robot and the robot identifier.

[0117] S204: According to the updated point cloud data of each robot and the point cloud data of each obstacle, the distance and the relative direction between each robot and each obstacle are calculated.

[0118] It should be noted that step S203 is similar to step S102 in embodiment one, and step S204 is similar to step S103 in embodiment one, which will not be described here.

[0119] The robot collision warning method provided in the embodiment improves the accuracy of the point cloud data of the robot by updating the point cloud data of the robot through the contact ground position of the robot, the pose of the robot and the substation point cloud data, and further improves the warning accuracy.

[0120] Figure 3 The flowchart of the robot collision warning method provided in the third embodiment of the present application is based on the above embodiments, and the present embodiment describes the case that the user controls the movement of the robot through the computer. As shown in the figure, Figure 3 The robot collision warning method specifically includes the following steps:

[0121] S301: Obtain user manipulation data.

[0122] In this step, the computer can display a robot working scene image, and when the user wants to manipulate the robot to go to a specific position, the user can click the robot and the target location in the robot working scene image in sequence, and the computer can generate user manipulation data, which includes data arranged in sequence. The first data is the position of the clicked robot image, and the second data is the position of the clicked target location image.

[0123] S302: Determine the clicked robot point cloud data and the target location according to the point cloud data corresponding to each pixel point in the robot working scene image, the clicked robot image position, and the clicked target location image position.

[0124] In this step, after the computer obtains the user operation data, the clicked robot point cloud data and the target location are determined according to the point cloud data corresponding to each pixel point in the robot working scene image, the clicked robot image position, and the clicked target location image position.

[0125] The pixel point of the clicked robot image position in the robot working scene image is determined, and then the point cloud data corresponding to the pixel point is determined to obtain the clicked robot point cloud data.

[0126] The pixel point of the clicked target location image position in the robot working scene image is determined, and then the point cloud data corresponding to the pixel point is determined. The average value of the three-dimensional coordinates in these point cloud data is taken as the target location.

[0127] S303: Determine the target robot identifier according to the clicked robot point cloud data, the point cloud data of each robot, and the robot identifier of each robot.

[0128] In this step, after the computer obtains the clicked robot point cloud data and the target location, in order to determine which robot needs to be controlled to move, the target robot identifier needs to be determined according to the clicked robot point cloud data, the point cloud data of each robot, and the robot identifier of each robot.

[0129] For each robot, if the point cloud data of the robot includes the clicked robot point cloud data, the robot identifier of the robot is taken as the target robot identifier.

[0130] It should be noted that if the point cloud data of each robot does not include the clicked robot point cloud data, an operation failure prompt message is output.

[0131] S304: Control the robot corresponding to the target robot identifier to move to the target location.

[0132] In this step, after the computer determines the target robot identifier, the robot corresponding to the target robot identifier is controlled to move to the target location.

[0133] The computer sends a movement request to the robot corresponding to the target robot identifier, and the movement request includes the target location. After receiving the movement request, the robot moves to the target location.

[0134] It should be noted that the user can also control the orientation of the robot. After the user clicks the robot and the target location in the robot working scene image in turn, the user can also click the object that needs to be oriented by the robot. The computer can generate user manipulation data, which includes data arranged in order. The first data is the position of the clicked robot image, the second data is the position of the clicked target location image, and the third data is the position of the clicked object image.

[0135] The computer determines the object point cloud position according to the point cloud data corresponding to each pixel point in the robot working scene image and the clicked object image position. The pixel point in the robot working scene image that clicks the object image position is determined, and then the point cloud data corresponding to the pixel point is determined. The average value of the three-dimensional coordinates in these point cloud data is taken as the object point cloud position.

[0136] Further, the target orientation is determined according to the object point cloud position and the target location. The direction from the target location to the object point cloud position is taken as the target orientation.

[0137] The computer controls the corresponding robot of the target robot identifier to identify the orientation as the target orientation. The computer sends an orientation request to the corresponding robot of the target robot identifier, and the orientation request includes the target orientation. After the robot receives the orientation request, it adjusts its own orientation to the target orientation.

[0138] The robot collision warning method provided in this embodiment can remotely control the movement and orientation of the robot through user manipulation data.

[0139] In the following, the training process of the first mapping model and the second mapping model is described through the fourth embodiment of the robot collision warning method provided in this application.

[0140] A training data is obtained from the training data set. For the first mapping model, the training data includes the robot working scene image, the substation point cloud data, and the internal and external parameters of the camera equipment. For the second mapping model, the training data includes the contact ground position of the robot, the pose of the robot, and the substation point cloud data.

[0141] The training data is input into the initial deep learning model to obtain training result data. For the first mapping model, the training result data is the point cloud data corresponding to each pixel point in the robot working scene image. For the second mapping model, the training result data is the point cloud data of the robot.

[0142] According to the training result data and the true value corresponding to the training data, the loss value is calculated. Then, the initial deep learning model is updated according to the loss value to obtain the trained model.

[0143] The number of training times is updated, that is, the number of training times is increased by one.

[0144] It is judged whether the updated training frequency is equal to a preset frequency. The preset frequency can be 500, 5000, 100000, etc., and the preset frequency is not limited by the embodiment of the application, and can be determined according to actual conditions.

[0145] If the updated training frequency is less than the preset frequency, the trained model is taken as a new initial deep learning model, and the above process is repeated until the updated training frequency is equal to the preset frequency. The latest trained model obtained is taken as a target model. For the training process of the first mapping model, the target model is the first mapping model. For the training process of the second mapping model, the target model is the second mapping model.

[0146] The robot collision warning method provided by the embodiment improves the accuracy of the first mapping model and the second mapping model by training the initial deep learning model according to the training data to obtain the first mapping model and the second mapping model.

[0147] The following is an apparatus embodiment of the application, which can be used to execute the method embodiments of the application. For details not disclosed in the apparatus embodiments of the application, refer to the method embodiments of the application.

[0148] Figure 4 The structure diagram of the robot collision warning device embodiment provided by the application is shown in FIG. 1. As shown in FIG. 1, the robot collision warning device 40 includes: Figure 4

[0149] The processing module 41 is configured to:

[0150] According to the robot working scene image captured by the camera device, the substation point cloud data, and the internal and external parameters of the camera device, the point cloud data of each robot and the robot identifier in the robot working scene image are determined, and the point cloud data of each obstacle is determined.

[0151] For each robot, according to the obtained historical point cloud data of the robot, the point cloud data and the robot identifier of the robot, the moving direction of the robot is determined.

[0152] According to the point cloud data of each robot and the point cloud data of each obstacle, the distance and the relative direction between each robot and each obstacle are calculated.

[0153] The warning module 42 is configured to perform collision warning according to the moving direction of each robot, the distance and the relative direction between each robot and each obstacle.

[0154] Further, the processing module 41 is specifically configured to:

[0155] ​input the robot working scene image, the substation point cloud data and the internal and external parameters of the camera equipment into a first mapping model to obtain point cloud data corresponding to each pixel point in the robot working scene image; the first mapping model is a deep learning model that is pre-trained to determine point cloud data corresponding to each pixel point in an image according to the image, the point cloud data and the internal and external parameters of the camera equipment;

[0156] perform target detection on the robot working scene image to obtain robot data of each robot and image positions of each obstacle, the robot data including the image positions and robot identifiers;

[0157] determine point cloud data of each robot and point cloud data of each obstacle according to the point cloud data corresponding to each pixel point in the robot working scene image, the image positions of each robot and the image positions of each obstacle.

[0158] Further, the early warning module 42 is specifically configured to:

[0159] for each robot and each obstacle, if the distance between the robot and the obstacle is less than a preset distance and the included angle between the relative direction of the robot and the obstacle and the moving direction of the robot belongs to a preset angle range, collision early warning is performed.

[0160] Further, the robot data of each robot further includes a pose, and before determining the moving direction of each robot according to the obtained robot historical point cloud data, the point cloud data of the robot and the robot identifier, the processing module 41 is further configured to:

[0161] for each robot, determine the contact ground position of the robot according to the image position of the robot;

[0162] for each robot, input the contact ground position of the robot, the pose of the robot and the substation point cloud data into a second mapping model to obtain updated point cloud data of the robot; the second mapping model is a deep learning model that is pre-trained to determine point cloud data corresponding to a robot according to the contact ground position, the pose and the point cloud data of the robot.

[0163] Further, the processing module 41 is specifically further configured to:

[0164] determine the moving direction of each robot according to the obtained robot historical point cloud data, the updated point cloud data of the robot and the robot identifier;

[0165] calculate the distance and the relative direction between each robot and each obstacle according to the updated point cloud data of each robot and the point cloud data of each obstacle.

[0166] The acquisition module 43 is configured to acquire user manipulation data, the user manipulation data including a clicked robot image position and a clicked target site image position.

[0167] Further, the processing module 41 is further configured to:

[0168] According to the point cloud data corresponding to each pixel point in the robot working scene image, the clicked robot image position, and the clicked target site image position, determine a clicked robot point cloud data and a target site.

[0169] According to the clicked robot point cloud data, the point cloud data of each robot, and the robot identifier of each robot, determine a target robot identifier.

[0170] Control a robot corresponding to the target robot identifier to move to the target site.

[0171] Further, if the user manipulation data further includes a clicked article image position, the processing module 41 is further configured to:

[0172] According to the point cloud data corresponding to each pixel point in the robot working scene image and the clicked article image position, determine an article point cloud position.

[0173] According to the article point cloud position and the target site, determine a target orientation.

[0174] Control the orientation of the robot corresponding to the target robot identifier to be the target orientation.

[0175] The robot collision warning device provided in the embodiment is used to execute the technical solutions in any of the method embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0176] Figure 5 A structural schematic diagram of an electronic device is provided in the present application. As shown in the figure, Figure 5 The electronic device 50 includes:

[0177] A processor 51, a memory 52, and a communication interface 53.

[0178] The memory 52 is configured to store executable instructions of the processor 51.

[0179] The processor 51 is configured to execute the technical solutions in any of the method embodiments by executing the executable instructions.

[0180] Optionally, the memory 52 can be independent or integrated with the processor 51.

[0181] Optionally, when the memory 52 is a device independent of the processor 51, the electronic device 50 can further include:

[0182] The bus 54, the memory 52 and the communication interface 53 are connected with the processor 51 through the bus 54 and complete communication with each other, and the communication interface 53 is used for communication with other devices.

[0183] Optionally, the communication interface 53 can be implemented by a transceiver. The communication interface is used for realizing communication between the database access device and other devices (for example, a client, a read-write library and a read-only library). The memory can include a random access memory (RAM) and can also include a non-volatile memory, for example, at least one disk memory.

[0184] The bus 54 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is represented in the figure, but it does not mean that there is only one bus or only one type of bus.

[0185] The processor described above can be a general-purpose processor, including a central processing unit CPU, a network processor (NP), etc.; can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0186] The electronic device is used for executing the technical solutions in any of the preceding method embodiments, and the implementation principles and technical effects are similar, and thus will not be described herein.

[0187] The embodiment of the present application further provides a readable storage medium, which has a computer program stored thereon, and the computer program is executed by a processor to implement the technical solutions provided by any of the preceding method embodiments.

[0188] The embodiment of the present application further provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the technical solutions provided by any of the preceding method embodiments.

[0189] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the foregoing storage medium includes various storage media that can store program codes, such as ROM, RAM, magnetic disk or optical disk.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A robot collision warning method, characterized by, The method comprises the following steps: inputting the acquired robot working scene image shot by a camera, substation point cloud data and internal and external parameters of the camera into a first mapping model to obtain point cloud data corresponding to each pixel point in the robot working scene image; the first mapping model is a deep learning model for determining point cloud data corresponding to each pixel point in an image according to the image, point cloud data and internal and external parameters of the camera; performing target detection on the robot working scene image to obtain robot data of each robot and image positions of each obstacle; the robot data comprises the image positions and robot identification; determining point cloud data of each robot and point cloud data of each obstacle according to the point cloud data corresponding to each pixel point in the robot working scene image, the image positions of each robot and the image positions of each obstacle; for each robot, determining a contact ground position of the robot according to the image position of the robot; for each robot, inputting the contact ground position of the robot, the posture of the robot and the substation point cloud data into a second mapping model to obtain updated point cloud data of the robot; the second mapping model is a deep learning model for determining point cloud data corresponding to a robot according to the contact ground position, the posture and the point cloud data of the robot; for each robot, determining a moving direction of the robot according to the acquired robot historical point cloud data, the point cloud data of the robot and the robot identification; calculating the distance and relative direction between each robot and each obstacle according to the point cloud data of each robot and the point cloud data of each obstacle; performing collision warning according to the moving direction of each robot, the distance and relative direction between each robot and each obstacle.

2. The method of claim 1, wherein, The collision warning according to the moving direction of each robot, the distance and relative direction between each robot and each obstacle comprises: for each robot and each obstacle, if the distance between the robot and the obstacle is less than a preset distance, and the included angle between the relative direction of the robot and the obstacle and the moving direction of the robot belongs to a preset angle range, collision warning is performed.

3. The method of claim 1, wherein, The determination of the moving direction of the robot according to the acquired robot historical point cloud data, the point cloud data of the robot and the robot identification comprises: determining the moving direction of the robot according to the acquired robot historical point cloud data, the updated point cloud data of the robot and the robot identification; The calculation of the distance and relative direction between each robot and each obstacle according to the point cloud data of each robot and the point cloud data of each obstacle comprises: calculating the distance and relative direction between each robot and each obstacle according to the updated point cloud data of each robot and the point cloud data of each obstacle.

4. The method of claim 1, wherein, The method further comprises: acquiring user manipulation data, the user manipulation data comprising clicked robot image positions and clicked target location image positions; According to the point cloud data corresponding to each pixel point in the robot working scene image, the click robot image position and the click target location image position, determine the click robot point cloud data and the target location; According to the click robot point cloud data, the point cloud data of each robot and the robot identifier of each robot, determine the target robot identifier; Control the robot corresponding to the target robot identifier to move to the target location.

5. The method of claim 4, wherein, If the user operation data further includes a click object image position, the method further includes: According to the point cloud data corresponding to each pixel point in the robot working scene image and the click object image position, determine the object point cloud position; According to the object point cloud position and the target location, determine the target orientation; Control the orientation of the robot corresponding to the target robot identifier to be the target orientation.

6. A robot collision warning device characterized by comprising: Comprise: The processing module is used to: Input the acquired robot working scene image shot by the camera device, the substation point cloud data and the internal and external parameters of the camera device into the first mapping model to obtain the point cloud data corresponding to each pixel point in the robot working scene image; the first mapping model is a pre-trained deep learning model for determining the point cloud data corresponding to each pixel point in the image according to the image, the point cloud data and the internal and external parameters of the camera device; Perform target detection on the robot working scene image to obtain robot data of each robot and image positions of each obstacle, the robot data including image positions and robot identifiers; According to the point cloud data corresponding to each pixel point in the robot working scene image, the image positions of each robot and the image positions of each obstacle, determine the point cloud data of each robot and the point cloud data of each obstacle; For each robot, determine the contact ground position of the robot according to the image position of the robot; For each robot, input the contact ground position of the robot, the pose of the robot and the substation point cloud data into the second mapping model to obtain the updated point cloud data of the robot; the second mapping model is a pre-trained deep learning model for determining the point cloud data corresponding to the robot according to the contact ground position, the pose and the point cloud data of the robot; For each robot, determine the moving direction of the robot according to the acquired robot historical point cloud data, the point cloud data of the robot and the robot identifier; According to the point cloud data of each robot and the point cloud data of each obstacle, calculate the distance and relative direction between each robot and each obstacle; The warning module is used to perform collision warning according to the moving direction of each robot, the distance and relative direction between each robot and each obstacle.

7. An electronic device, comprising: Comprise: A processor, a memory and a communication interface; The memory is used to store executable instructions of the processor; Wherein, the processor is configured to execute the executable instructions to perform the robot collision warning method in any one of claims 1 to 5.

8. A readable storage medium, having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the robot collision warning method in any one of claims 1 to 5.

9. A computer program product, characterised in that, A computer program product comprising a computer program which, when executed by a processor, is configured to implement the robot collision warning method of any one of claims 1 to 5.

Images