Dynamic falling monitoring method and device of mechanical arm, storage medium and equipment

Abstract

The embodiment of the invention relates to a falling dynamic monitoring method and device of a mechanical arm, a storage medium and equipment, and the method comprises the steps that under the condition that the mechanical arm is in a motion state, a plurality of monitoring images sent by an image collection device are received in response to a power-off signal of the mechanical arm, the multiple monitoring images are obtained by performing image acquisition on an end effector of the mechanical arm in the falling motion process of the mechanical arm through an image acquisition device, and a detection ball is configured on the end effector; for each monitoring image, determining a center coordinate of a detection ball in the monitoring image; and according to the center coordinates of the detection balls in the multiple monitoring images, the falling dynamic information of the mechanical arm is determined. According to the embodiment of the invention, the cost can be reduced, and the operation is simple and does not need to be performed by professionals, so that the use and popularization are convenient.

Description

Technical Field

[0001] The disclosed embodiments relate to the technical field of surgical robots, and in particular to a method, device, storage medium, and equipment for dynamically monitoring the fall of a robotic arm. Background Art

[0002] In order to ensure the quality of surgical robots, surgical robots usually undergo a series of tests before they are officially sold on the market. Among them, the robot arm drop detection is an important item that affects whether the surgical robot can be used normally. The robot arm drop detection includes the static position change and dynamic position change of the robot arm drop. Static position change refers to the change in data measured statically before and after the power failure of the robot arm, that is, the robot arm keeps its current position and uses three coordinates to measure a position. After the power failure, the robot arm falls. When the falling action stops, the three coordinates are used to measure a position again. The position change before and after the power failure is calculated to obtain the static position change of the robot arm. Dynamic position change refers to the dynamic position change of the robot arm during the falling process after the robot arm is powered off. Since the robot arm will be in a repeated movement of falling, rebounding, falling, rebounding... in a short period of time after the power failure, it is necessary to be able to measure the dynamic position change of the end effector of the robot arm.

[0003] At present, the monitoring method of dynamic position change is relatively simple, and laser trackers are mostly used to track the position of the end effector of the robot arm. However, laser trackers are expensive and costly to use. In addition, the laser trackers need to be operated by professionals, which makes them difficult to use and promote. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a method, device, storage medium and equipment for dynamic monitoring of the fall of a robotic arm.

[0005] A first aspect of an embodiment of the present disclosure provides a method for dynamically monitoring a fall of a robotic arm, the method comprising:

[0006] When the robot arm is in motion, in response to a power-off signal of the robot arm, receiving a plurality of monitoring images sent by an image acquisition device, wherein the plurality of monitoring images are obtained by the image acquisition device performing image acquisition on an end effector of the robot arm during a falling motion of the robot arm, and a detection ball is arranged on the end effector;

[0007] For each monitoring image, determining the center coordinates of the detection ball in the monitoring image;

[0008] The falling dynamic information of the robotic arm is determined according to the center coordinates of the detection ball in the multiple monitoring images.

[0009] A second aspect of an embodiment of the present disclosure provides a dynamic fall monitoring device for a mechanical arm, the device comprising:

[0010] a first receiving module, configured to receive a plurality of monitoring images sent by an image acquisition device in response to a power-off signal of the robotic arm when the robotic arm is in motion, wherein the plurality of monitoring images are obtained by the image acquisition device performing image acquisition on an end effector of the robotic arm during a falling motion of the robotic arm, and a detection ball is disposed on the end effector;

[0011] A first determination module is used to determine, for each monitoring image, the center coordinates of the detection ball in the monitoring image;

[0012] The second determination module is used to determine the falling dynamic information of the robotic arm according to the center coordinates of the detection ball in the multiple monitoring images.

[0013] A third aspect of an embodiment of the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method of the first aspect described above can be implemented.

[0014] A fourth aspect of the disclosed embodiments provides a dynamic monitoring device for a mechanical arm falling, comprising: a power failure signal capturing device, an image acquisition device, a detection ball, and a monitoring device;

[0015] The power-off signal capturing device is used to capture the power-off signal of the robot arm;

[0016] The image acquisition device is used to acquire images of the end effector of the robotic arm during the falling motion of the robotic arm to obtain a plurality of monitoring images;

[0017] The detection ball is configured on the end effector;

[0018] The monitoring device is connected to the power-off signal capturing device and the image acquisition device respectively, and the monitoring device is used to execute the method of the first aspect mentioned above.

[0019] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0020] The disclosed embodiment can receive multiple monitoring images sent by the image acquisition device in response to the power-off signal of the robotic arm when the robotic arm is in motion, wherein the multiple monitoring images are obtained by the image acquisition device performing image acquisition on the end effector of the robotic arm during the falling motion of the robotic arm, and a detection ball is configured on the end effector; for each monitoring image, the center coordinates of the detection ball in the monitoring image are determined; and the falling dynamic information of the robotic arm is determined according to the center coordinates of the detection ball in the multiple monitoring images. It can be seen that by adopting the above technical solution, in the process of monitoring the falling dynamics of the robotic arm, there is no need to use a laser tracker, and the image acquisition device can be used to perform image acquisition on the end effector during the falling motion of the robotic arm, and then the falling dynamic information of the robotic arm can be determined according to the multiple monitoring images acquired by the image acquisition device, so that the cost can be reduced, and because the operation is simple and does not require professional personnel to operate, it is easy to use and promote. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 is a flow chart of a method for dynamically monitoring the fall of a robotic arm provided by an embodiment of the present disclosure;

[0024] Figure 2 It is a structural schematic diagram of a dynamic fall monitoring device for a mechanical arm provided in an embodiment of the present disclosure;

[0025] Figure 3 is a structural schematic diagram of another mechanical arm fall dynamic monitoring device provided by an embodiment of the present disclosure;

[0026] Figure 4 is a structural schematic diagram of an image acquisition device provided by an embodiment of the present disclosure;

[0027] Figure 5 is a structural schematic diagram of another image acquisition device provided by an embodiment of the present disclosure;

[0028] Figure 6 is a structural schematic diagram of a surgical robot provided by an embodiment of the present disclosure;

[0029] Figure 7is a schematic diagram of a structure in which a detection ball is installed on an end effector provided by an embodiment of the present disclosure;

[0030] Figure 8 is a structural schematic diagram of a backlight panel provided by an embodiment of the present disclosure;

[0031] Fig. 9 is a schematic diagram of a scenario of dynamically monitoring a robot arm falling provided by an embodiment of the present disclosure;

[0032] Fig.10 It is a structural schematic diagram of a dynamic fall monitoring device for a mechanical arm provided by an embodiment of the present disclosure;

[0033] Fig.11 It is a structural schematic diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0035] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0036] Figure 1 It is a flow chart of a method for dynamically monitoring the fall of a robotic arm provided in an embodiment of the present disclosure. The method can be executed by a dynamic monitoring device for the fall of a robotic arm. The dynamic monitoring device for the fall of a robotic arm can be implemented by software and / or hardware. Exemplarily, the dynamic monitoring device for the fall of a robotic arm can be understood as a monitoring device in the dynamic monitoring equipment for the fall of a robotic arm. In order to clearly illustrate the method for dynamically monitoring the fall of a robotic arm provided in an embodiment of the present disclosure, the dynamic monitoring equipment for the fall of a robotic arm will be described in detail below. Figure 2 It is a structural schematic diagram of a dynamic fall monitoring device for a robotic arm provided in an embodiment of the present disclosure. Figure 3 It is a structural schematic diagram of another mechanical arm fall dynamic monitoring device provided in an embodiment of the present disclosure. Figure 4 It is a structural schematic diagram of an image acquisition device provided in an embodiment of the present disclosure. Figure 5 is a schematic diagram of the structure of another image acquisition device provided by the embodiment of the present disclosure. Figure 2 and Figure 3 This is a schematic diagram of the structure of the dynamic fall monitoring device of the same robotic arm at different viewing angles. Figure 4 and Figure 5It is a schematic diagram of the structure of the same image acquisition device at different viewing angles. Figure 2-Figure 5 As shown, the dynamic monitoring device for the fall of the robot arm includes: a power-off signal capturing device, an image acquisition device 110, a detection ball 120 and a monitoring device;

[0037] The power-off signal capturing device is used to capture the power-off signal of the robot arm;

[0038] The image acquisition device 110 is used to acquire images of the end effector of the robotic arm during the falling motion of the robotic arm to obtain multiple monitoring images;

[0039] The detection ball is configured on the end effector;

[0040] The monitoring device is used to execute the method described in any embodiment of the present disclosure.

[0041] Specifically, the power-off signal capturing device can be connected to the "control device for power-off control of the robot arm on the surgical robot" (wired connection or wireless connection) to capture the power-off signal of the robot arm. Figure 6 Schematic diagram of the structure of a surgical robot provided by an embodiment of the present disclosure. Figure 6 As shown, the surgical robot includes a robotic arm 210 and a trolley 220. The trolley 220 provides the surgical robot with necessary mobility and support functions, wherein the support functions include, for example, robotic arm motion control, robotic arm power-on control, and robotic arm power-off control, etc. Therefore, the power-off signal capture device can be connected to the trolley 220 of the surgical robot to capture the power-off signal of the robotic arm 210.

[0042] Specifically, when the robot arm 210 is in motion (e.g., repeatedly moving up and down), if the robot arm 210 is powered off (or stops suddenly), the robot arm 210 will fall. At this time, the image acquisition device 110 starts to acquire images of the end effector 211 of the robot arm 210, thereby obtaining multiple monitoring images. The falling motion means that after the power is off, the robot arm 210 will be in a repeated motion of falling, rebounding, falling, rebounding, etc. in a short period of time. In some examples, the image acquisition device 110 can acquire images of the end effector 211 of the robot arm 210 in response to receiving the first acquisition signal sent by the power-off signal capture device. Of course, in other examples, the dynamic monitoring device for the fall of the robot arm also includes a control board, and the control board can control the image acquisition device 110 to acquire images of the end effector 211 of the robot arm 210 in response to receiving the second acquisition signal sent by the power-off signal capture device. This is not limited to this.

[0043] Optionally, the resolution of the image acquisition device 110 is greater than or equal to 1280×800, and the image acquisition frequency is greater than or equal to 76000 fps. In this way, the motion state of the end effector 211 in a short period of time can be captured and recorded.

[0044] Exemplarily, the image acquisition device 110 includes an ultra-high-speed camera with a resolution of 1280×800 and a refresh rate of up to 76,000 fps.

[0045] Specifically, the detection ball is disposed on the end effector 211 , so that the falling dynamic information of the robot arm 210 can be determined later by analyzing the center coordinates of the detection ball in the monitoring image.

[0046] Optionally, the end effector 211 and the detection ball are mechanically connected.

[0047] Specifically, when it is necessary to perform dynamic monitoring of the fall of the robot arm 210, the detection ball can be installed on the end effector 211. For example, the detection ball can be installed on the end effector 211 using screws, clamps or other fasteners to mechanically connect the two. When the dynamic monitoring of the fall of the robot arm 210 is completed, the detection ball can be removed from the end effector 211.

[0048] For example, Figure 7 2 is a schematic diagram of a structure in which a detection ball is installed on an end effector 211 according to an embodiment of the present disclosure. Figure 6 and Figure 7 As shown, the detection ball 120 is installed on the end effector 211 , and the diameter of the ball is about 30 mm (for example, 30 mm). The detection ball 120 needs to be customized to match the end effector 211 .

[0049] It is understandable that in the prior art, the reflective ball of the laser tracker is usually glued to the end effector 211, which may cause damage to the surface of the end effector 211 when the reflective ball is removed. The embodiment of the present disclosure provides a mechanical connection between the end effector 211 and the detection ball 120, rather than gluing the detection ball 120 to the end effector 211. In this way, damage to the surface of the end effector 211 can be avoided.

[0050] Specifically, the monitoring device may be implemented by software and / or hardware. Exemplarily, the monitoring device may include a host computer, etc., but is not limited thereto.

[0051] Optionally, the dynamic monitoring of the fall of the robotic arm also includes: a signal transmission device, used to transmit a plurality of monitoring images acquired by the image acquisition device to the monitoring device.

[0052] Optionally, the dynamic monitoring of the fall of the robot arm further includes: a whole machine frame, the whole machine frame includes a support portion 130 and a plurality of adjustment legs 140;

[0053] The image acquisition device 110 and the power-off signal capture device are both arranged on the support part 130. The support part 130 is also provided with a horizontal leveling rod 160 and a vertical leveling rod 150. The horizontal leveling rod 160 is used to indicate the horizontality of the support part 130 in the horizontal direction, and the vertical leveling rod 150 is used to indicate the verticality of the support part 130 in the vertical direction. Among them, the plane formed by the intersection of the horizontal direction and the vertical direction is perpendicular to the ground, the horizontal direction is parallel to the ground, and the vertical direction is perpendicular to the ground.

[0054] A plurality of adjustment legs 140 are connected to the support portion 130 , and the plurality of adjustment legs 140 are used to adjust the height and level of the support portion 130 .

[0055] Specifically, the horizontal leveling ruler 160 is used to ensure that the support portion 130 is in a horizontal state in the horizontal direction, and the vertical leveling ruler 150 is used to ensure that the support portion 130 is in a vertical state in the vertical direction.

[0056] Specifically, a plurality of adjustment feet 140 are disposed below the support portion 130. The height of the support portion 130 can be adjusted through the plurality of adjustment feet 140 to adapt to different working environments or operator needs. In conjunction with the horizontal level 160 and the vertical level 150, the support portion 130 can be adjusted by fine-tuning the height of the adjustment feet 140 to ensure that the support portion 130 is horizontal in the horizontal direction and vertical in the vertical direction.

[0057] For example, see Figure 2 and Figure 3 An ultra-high-speed camera is installed on one side of the support part 130, and four adjustment feet 140 are respectively located at the four corners of the bottom of the support part 130. A horizontal level ruler 160 and a vertical level ruler 150 are installed in the middle of the support part 130. By adjusting the adjustment feet 140 and observing the horizontal level ruler 160 and the vertical level ruler 150, the support part 130 is controlled to be in a horizontal state in the horizontal direction and in a vertical state in the vertical direction. Of course, the dynamic monitoring device for the fall of the robot arm 210 can also include a power supply device, etc. The power supply device, the power-off signal capture device, the control board and the signal transmission device can be located on the support part 130 and below the ultra-high-speed camera. The power-off signal capture device captures the power-off signal when the robot arm 210 is powered off, and feeds back the second acquisition signal to the control board. After receiving the second acquisition signal fed back by the power-off signal capture device, the control board starts to collect the monitoring image taken by the ultra-high-speed camera, and the signal transmission device transmits the monitoring image to the monitoring device.

[0058] It is understandable that the dynamic position change refers to the amount of dynamic position change of the robotic arm 210 during the falling process after the robotic arm 210 is powered off. Since the robotic arm 210 will be in a repeated motion of falling, rebounding, falling, rebounding, etc. in a short period of time after the power is off, it is necessary to measure the dynamic coordinate data of the end effector 211 of the robotic arm 210 during the falling process in order to calculate the dynamic position change of the robotic arm 210. The dynamic position change mainly focuses on the position change of the robotic arm 210 in the vertical direction (that is, perpendicular to the ground). By setting a plurality of adjustment feet 140, a horizontal leveling ruler 160 and a vertical leveling ruler 150, it is convenient for the user to adjust the image acquisition device 110 to a horizontal state in the horizontal direction and a vertical state in the vertical direction, so as to accurately determine the dynamic position change of the robotic arm falling according to the center coordinates of the detection ball 120 in the monitoring image.

[0059] Optionally, the dynamic fall monitoring of the robot arm 210 further includes: a lens light source 111 and a backlight panel 170;

[0060] The backlight panel 170 is disposed on the whole machine frame and is disposed opposite to the image acquisition device 110. The backlight panel 170 is provided with a backlight source 171 and an adjustment component 172. The light emitting side of the backlight source 171 faces the image acquisition device 110. The adjustment component 172 is used to adjust the light emitting brightness of the backlight source 171.

[0061] The lens light source 111 is disposed on the image acquisition device 110 , and the light emitting side of the lens light source 111 faces the backlight panel 170 . The lens light source 111 includes a plurality of light source partitions, and each light source partition is independently controlled.

[0062] For example, see Figure 2 and Figure 3 An ultra-high-speed camera is installed on one side of the support portion 130, and a backlight plate 170 is installed on the other side, so that the ultra-high-speed camera and the backlight plate 170 are arranged opposite to each other.

[0063] Specifically, the backlight panel 170 is mainly used as a background panel during the image acquisition process to isolate the interference of surrounding objects. Figure 8 is a schematic diagram of the structure of a backlight panel provided by an embodiment of the present disclosure, such as Figure 8 As shown, the backlight panel 170 has a built-in backlight light source 171, and the brightness can be adjusted by the adjustment component 172 according to the needs. When the backlight light source 171 is turned on, the monitoring image captured by the image acquisition device 110 is a black and white image. When the backlight light source 171 is turned off, the color of the monitoring image captured by the image acquisition device 110 is close to the actual color.

[0064] For example, see Figure 4The lens light source 111 has eight light source partitions, and each light source partition can be controlled separately.

[0065] It can be understood that by providing the lens light source 111 and the backlight panel 170 , the quality of the detection image can be significantly improved, helping to achieve more accurate detection and identification of the detection ball 120 .

[0066] like Figure 1 As shown, the method provided in this embodiment includes the following steps:

[0067] S110. When the robot arm is in motion, in response to a power-off signal of the robot arm, multiple monitoring images sent by an image acquisition device are received, wherein the multiple monitoring images are obtained by the image acquisition device capturing images of the end effector of the robot arm during the falling motion of the robot arm, and a detection ball is configured on the end effector.

[0068] In the disclosed embodiment, Fig. 9 FIG. 1 is a schematic diagram of a scenario for dynamically monitoring a robot arm falling provided by an embodiment of the present disclosure. Fig. 9 As shown, before the robot 210 is dynamically monitored for falling, some preparations need to be made: the surgical robot is initially positioned, the dynamic monitoring device of the robot's falling is placed in the common surgical area of ​​the surgical robot, and the dynamic monitoring device of the robot's falling is placed stably by adjusting the adjustment feet and observing the horizontal level ruler and the vertical level ruler (i.e., it is horizontal in the horizontal direction and vertical in the vertical direction). Wiring operations are also required: ① connect the power cord; ② connect the power-off signal capture device to the emergency stop / power-off response interface of the surgical robot; ③ connect the external data transmission line, which can be connected to the robot body computer (in this case, the robot body computer is used as a monitoring device), or to other host computers (in this case, the host computer is used as a monitoring device). The detection ball 120 should also be installed on the end effector of the robot 210, and the detection ball 120 should be dragged to the center position of the dynamic monitoring device of the falling of the robot 210 (i.e., the center of the field of view of the image acquisition device). When dragging, the monitoring device can be checked to determine the position of the detection ball 120, and if the position is not suitable, appropriate adjustments should be made. Then, the program of repeated up and down movement pre-set by the robot arm 210 is run to ensure that the detection ball 120 on the end effector 211 of the robot arm 210 can always be within the detection range (that is, always within the field of view of the image acquisition device 110). After the above preparations are completed, the robot arm 210 is powered off or stopped urgently, so that the image acquisition device 110 starts to acquire images of the end effector 211 to obtain monitoring images, and transmit the monitoring images to the monitoring device.

[0069] S120: For each monitoring image, determine the center coordinates of the detection ball in the monitoring image.

[0070] In the embodiment of the present disclosure, the monitoring device determines the center coordinates of the detection ball in each monitoring image. It should be noted that the monitoring device can determine the center coordinates of the detection ball in the monitoring image while receiving the monitoring image, or can determine the center coordinates of the detection ball in the monitoring image for each monitoring image after receiving all monitoring images, and the present disclosure does not limit this.

[0071] It can be understood that the shape of the detection ball is a sphere, and the sphere has the same geometric properties in any posture. The center of the sphere is not affected by posture changes. Therefore, the center coordinates of the detection ball in the monitoring image can be calculated and the center coordinates of the detection ball can be used to represent the position of the detection ball, thereby improving the accuracy of the falling dynamic information.

[0072] In some embodiments, S120 includes: for each monitoring image, identifying a circular outline of the detection ball from the monitoring image, and determining the center coordinates of the detection ball according to the circular outline.

[0073] Specifically, the circular contour of the detection ball can be identified from the monitoring image and the center coordinates of the detection ball (ie, the center coordinates of the circle) can be calculated based on a shape fitting method or a contour extraction method, but is not limited thereto.

[0074] It can be understood that by first identifying the circular outline of the detection ball and then determining the center coordinates of the detection ball based on the circular outline, the position of the detection ball can be determined more accurately, especially in a complex background. This method can reduce errors caused by noise. Moreover, compared to directly searching for the detection ball on the entire monitoring image, calculating the center of the circle after detecting the circular outline is usually simpler and more efficient.

[0075] In other embodiments, S120 includes: inputting the monitoring image into a trained first neural network model, and receiving the center coordinates of the detection ball output by the first neural network model.

[0076] S130. Determine the falling dynamic information of the robot arm according to the center coordinates of the detection ball in the multiple monitoring images.

[0077] In the disclosed embodiment, the monitoring device determines the dynamic information of the fall of the robot arm according to the center coordinates of the detection ball in the multiple monitoring images, wherein the dynamic information of the fall is the dynamic position change, and the dynamic position change may include the position change between the highest point coordinate and the lowest point coordinate of the end effector (i.e., the vertical position difference), the fall data of the end effector and / or the motion trajectory of the end effector. Of course, after determining the dynamic information of the fall of the robot arm, the dynamic information of the fall may also be output in the form of a report.

[0078] In some embodiments, S130 includes: selecting the highest point coordinate corresponding to when the end effector is at the highest point in the vertical direction and the lowest point coordinate corresponding to when the end effector is at the lowest point from the center coordinates of the detection ball in the plurality of monitoring images, and calculating the vertical position difference between the highest point coordinate and the lowest point coordinate;

[0079] Acquire the internal and external parameters of the image acquisition device and determine the drop data of the end effector in the vertical direction based on the internal and external parameters of the image acquisition device and the vertical position difference; or acquire the depth information of the detection ball and determine the drop data of the end effector in the vertical direction based on the depth information and the vertical position difference. Specifically, Figure 4 As shown, for each monitoring image, the circular contour corresponding to the monitoring image has a central coordinate (x, y), wherein the y coordinate is used to characterize the position of the detection ball in the vertical direction (i.e., perpendicular to the ground). According to the acquisition time of multiple monitoring images, the central coordinates (x, y) of the circular contours corresponding to the multiple monitoring images can be used to determine the change of the central coordinates in the vertical direction over time. Based on the change, the highest point coordinate corresponding to when the detection ball is at the highest point and the lowest point coordinate corresponding to when the detection ball is at the lowest point can be determined. The difference between the highest point coordinate and the lowest point coordinate in the vertical direction can be calculated to obtain the vertical position difference.

[0080] Specifically, the falling data is to detect the change of the three-dimensional coordinates of the ball during the falling motion.

[0081] Specifically, the internal and external parameters of the image acquisition device are the internal and external parameters of the image acquisition device. It should be noted that the internal and external parameters of the image acquisition device can be determined in any manner known to those skilled in the art. Exemplarily, before the falling dynamics of the robot arm are monitored, the internal parameters of the image acquisition device can be determined in advance, such as placing a checkerboard calibration plate (with black and white grids alternating and with a certain length and width) between the lens and the background plate to capture multiple images, and then determining the internal parameters of the image acquisition device based on the multiple images, but it is not limited to this.

[0082] Specifically, when the image acquisition device cannot obtain the depth information of the detection ball (for example, when the image acquisition device is an ultra-high-speed camera), the internal and external parameters of the image acquisition device can be obtained, and then the drop data of the end effector in the vertical direction can be determined based on the internal and external parameters of the image acquisition device and the vertical position difference.

[0083] Specifically, when the image acquisition device can not only obtain monitoring images but also obtain depth information of the detection ball (for example, when the image acquisition device is a deep ultra-high-speed camera), the drop data of the end effector in the vertical direction can be determined based on the depth information and the vertical position difference.

[0084] Optionally, S130 may further include: simulating a motion trajectory of the end effector according to the acquisition moments corresponding to the multiple monitoring images and the center coordinates of the detection balls in the multiple monitoring images.

[0085] Specifically, according to the acquisition time of multiple monitoring images, the two-dimensional motion trajectory of the detection ball can be drawn through the center coordinates (x, y) of the circular contours corresponding to the multiple monitoring images, or, according to the acquisition time of multiple monitoring images, the three-dimensional motion trajectory of the detection ball can be drawn through the center coordinates (x, y) of the circular contours corresponding to the multiple monitoring images and the internal and external parameters of the image acquisition device, or, according to the acquisition time of multiple monitoring images, the three-dimensional motion trajectory of the detection ball can be drawn through the center coordinates (x, y) of the circular contours corresponding to the multiple monitoring images and the depth information of each center coordinate (x, y).

[0086] It can be understood that by selecting the highest point coordinates and the lowest point coordinates and calculating the vertical position difference between the highest point coordinates and the lowest point coordinates, determining the drop data of the end effector in the vertical direction according to the vertical position difference, and drawing the motion trajectory according to the acquisition time and center coordinates corresponding to multiple monitoring images, the method of determining the dynamic information of the drop can have the advantages of small calculation amount and easy implementation.

[0087] In other embodiments, S120 includes: inputting the center coordinates of the ball detected from multiple monitoring images into a trained second neural network model, and receiving the falling dynamic information of the robotic arm output by the second neural network model.

[0088] According to the disclosed embodiments, in the process of monitoring the falling dynamics of the robotic arm, there is no need to use a laser tracker. Instead, an image acquisition device can be used to acquire images of the end effector during the falling motion of the robotic arm. Then, the falling dynamic information of the robotic arm can be determined based on a plurality of monitoring images acquired by the image acquisition device. In this way, the cost can be reduced. Furthermore, since the operation is simple and does not require professional operation, it is easy to use and promote.

[0089] Fig.10 Schematic diagram of a dynamic monitoring device for a mechanical arm falling provided by an embodiment of the present disclosure. Fig.10 As shown, the dynamic monitoring device 1000 for falling of the robot arm includes:

[0090] The first receiving module 1010 is used for receiving a plurality of monitoring images sent by an image acquisition device in response to a power-off signal of the robotic arm when the robotic arm is in motion, wherein the plurality of monitoring images are obtained by the image acquisition device performing image acquisition on an end effector of the robotic arm during a falling motion of the robotic arm, and a detection ball is arranged on the end effector;

[0091] A first determination module 1020 is used to determine, for each monitoring image, the center coordinates of the detection ball in the monitoring image;

[0092] The second determination module 1030 is used to determine the falling dynamic information of the robotic arm according to the center coordinates of the detection ball in the multiple monitoring images.

[0093] Optionally, the first determination module 1020 is specifically configured to identify, for each monitoring image, a circular outline of the detection ball from the monitoring image, and determine the center coordinates of the detection ball according to the circular outline.

[0094] Optionally, the falling dynamic information includes falling data of the end effector in the vertical direction, wherein the second determining module 1030 is specifically used to select the highest point coordinate corresponding to when the end effector is at the highest point in the vertical direction and the lowest point coordinate corresponding to when the end effector is at the lowest point from the center coordinates of the detection ball in the multiple monitoring images, and calculate the vertical position difference between the highest point coordinate and the lowest point coordinate;

[0095] Acquire the internal and external parameters of the image acquisition device and determine the drop data of the end effector in the vertical direction based on the internal and external parameters of the image acquisition device and the vertical position difference, or acquire the depth information of the detection ball and determine the drop data of the end effector in the vertical direction based on the depth information and the vertical position difference.

[0096] The device provided in this embodiment can execute the method of any of the above embodiments, and its execution method and beneficial effects are similar, which will not be repeated here.

[0097] An embodiment of the present disclosure further provides an electronic device, which includes: a memory, in which a computer program is stored; and a processor, for executing the computer program. When the computer program is executed by the processor, the method of any of the above embodiments can be implemented.

[0098] For example, Fig.11 Schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Fig.11 , which shows a schematic diagram of the structure of an electronic device 1100 suitable for implementing the embodiment of the present disclosure. The electronic device 1100 in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.11The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0099] like Fig.11 As shown, the electronic device 1100 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage device 1108 to a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the electronic device 1100 are also stored. The processing device 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0100] Typically, the following devices may be connected to the I / O interface 1105: an input device 1106 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1108 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1109. The communication device 1109 may allow the electronic device 1100 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.11 The electronic device 1100 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0101] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1109, or installed from a storage device 1108, or installed from a ROM 1102. When the computer program is executed by the processing device 1101, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0102] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0103] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0104] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0105] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: when the robotic arm is in motion, in response to a power-off signal of the robotic arm, receives multiple monitoring images sent by an image acquisition device, wherein the multiple monitoring images are obtained by the image acquisition device performing image acquisition on the end effector of the robotic arm during the falling motion of the robotic arm, and a detection ball is configured on the end effector; for each monitoring image, determines the center coordinates of the detection ball in the monitoring image; and determines the falling dynamic information of the robotic arm based on the center coordinates of the detection ball in the multiple monitoring images.

[0106] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0107] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0109] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0110] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] The embodiments of the present disclosure further provide a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method of any of the above embodiments can be implemented. The execution method and beneficial effects are similar and will not be repeated here.

[0112] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0113] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for dynamic monitoring of a robot arm falling, characterized in that: include: When the robot arm is in motion, in response to a power-off signal of the robot arm, receiving a plurality of monitoring images sent by an image acquisition device, wherein the plurality of monitoring images are obtained by the image acquisition device performing image acquisition on an end effector of the robot arm during a falling motion of the robot arm, and a detection ball is arranged on the end effector; For each monitoring image, determining the center coordinates of the detection ball in the monitoring image; The falling dynamic information of the robotic arm is determined according to the center coordinates of the detection ball in the multiple monitoring images.

2. The method according to claim 1, characterized in that The step of determining the center coordinates of the detection ball in each monitoring image includes: For each monitoring image, a circular outline of the detection ball is identified from the monitoring image, and the center coordinates of the detection ball are determined according to the circular outline.

3. The method according to claim 1, characterized in that The falling dynamic information includes falling data of the end effector in the vertical direction, wherein the determining the falling dynamic information of the robotic arm according to the center coordinates of the detection ball in the plurality of monitoring images includes: From the center coordinates of the detection sphere in the plurality of monitoring images, select the highest point coordinate corresponding to when the end effector is at the highest point in the vertical direction and the lowest point coordinate corresponding to when the end effector is at the lowest point, and calculate the vertical position difference between the highest point coordinate and the lowest point coordinate; Acquire the internal and external parameters of the image acquisition device and determine the drop data of the end effector in the vertical direction based on the internal and external parameters of the image acquisition device and the vertical position difference, or acquire the depth information of the detection ball and determine the drop data of the end effector in the vertical direction based on the depth information and the vertical position difference.

4. A dynamic monitoring device for a mechanical arm falling, characterized in that: include: a first receiving module, configured to receive a plurality of monitoring images sent by an image acquisition device in response to a power-off signal of the robotic arm when the robotic arm is in motion, wherein the plurality of monitoring images are obtained by the image acquisition device performing image acquisition on an end effector of the robotic arm during a falling motion of the robotic arm, and a detection ball is disposed on the end effector; A first determination module is used to determine, for each monitoring image, the center coordinates of the detection ball in the monitoring image; The second determination module is used to determine the falling dynamic information of the robotic arm according to the center coordinates of the detection ball in the multiple monitoring images.

5. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

6. A dynamic monitoring device for a mechanical arm falling, characterized in that: include: Power failure signal capture device, image acquisition device, detection ball and monitoring device; The power-off signal capturing device is used to capture the power-off signal of the robot arm; The image acquisition device is used to acquire images of the end effector of the robotic arm during the falling motion of the robotic arm to obtain a plurality of monitoring images; The detection ball is configured on the end effector; The monitoring device is used to perform the method according to any one of claims 1 to 3.

7. The device according to claim 6, characterized in that Also includes: A whole machine frame, the whole machine frame comprising a support portion and a plurality of adjustment legs; The image acquisition device and the power-off signal capture device are both arranged on the support portion, and a horizontal level ruler and a vertical level ruler are also arranged on the support portion, wherein the horizontal level ruler is used to indicate the horizontality of the support portion in the horizontal direction, and the vertical level ruler is used to indicate the verticality of the support portion in the vertical direction; The plurality of adjustment feet are connected to the support portion, and the plurality of adjustment feet are used to adjust the height and level of the support portion.

8. The device according to claim 7, characterized in that Also includes: Lens light source and backlight panel; The backlight panel is arranged on the whole machine frame and is arranged opposite to the image acquisition device. A backlight light source and an adjustment component are arranged on the backlight panel. The light emitting side of the backlight light source faces the image acquisition device. The adjustment component is used to adjust the light emitting brightness of the backlight light source. The lens light source is arranged on the image acquisition device, and the light emitting side of the lens light source faces the backlight panel. The lens light source includes a plurality of light source partitions, and each light source partition is independently controlled.

9. The device according to claim 6, characterized in that The end effector is mechanically connected to the detection ball.

10. The device according to claim 6, characterized in that The resolution of the image acquisition device is greater than or equal to 1280×800, and the image acquisition frequency is greater than or equal to 76000fps.

Images