Method for determining position of operating object, robot and automation system

Abstract

The present application discloses a method for determining the position of an operation object, a robot, a robot system and an automation system, and belongs to the field of automation technology. The method for determining the position of an operation object disclosed in the embodiment of the present application arranges reference marks on the operation subject as reference positions, and obtains a global image containing reference marks taken by a camera on the robot, so that the user coordinate system of the operation subject and the tool coordinate system of the robot can be associated through the camera, thereby converting the position of the operation object in the user coordinate system into the position of the operation object in the tool coordinate system of the robot, so that the robot can subsequently move to the operating position corresponding to the operation object according to the position of the operation object in the tool coordinate system, and operate the operation object.

Description

Technical Field

[0001] The present application relates to the field of automation technology, and in particular to a method for determining the position of an operation object, a robot, a robot system and an automation system. Background Art

[0002] In the context of Industry 5.0, the intelligent transformation of optical distribution frame (ODF) rooms is imperative. The intelligence of ODF rooms, that is, the automation of ODF room operations, means that daily operations in ODF rooms, such as fiber plugging and unplugging, fiber port cleaning, and fiber port testing, are all performed by robots instead of manual labor.

[0003] Before the robot operates an operating object such as a fiber optic port, it needs to first determine its relative position to the operating object. Only then can the robot determine the moving direction and distance, and then move to the operating position to operate the operating object. Summary of the invention

[0004] The embodiments of the present application provide a method for determining the position of an operation object, a robot, a robot system, and an automation system. The method for determining the position of an operation object can be applied to a robot, so that the robot can determine the position of the operation object relative to the center point of a tool. The technical solutions of the method for determining the position of an operation object, the robot, the robot system, and the automation system are as follows:

[0005] In a first aspect, a method for determining a position of an operation object is provided, the method being applied in a robot, the method comprising:

[0006] Acquire a global image of an operating subject taken by a camera located on the robot, wherein the operating subject is a subject where an operating object of the robot is located, a reference mark and at least one operating object are distributed on the same side of the operating subject, and the global image is an image including the reference mark; acquire the positions of a target operating object and the reference mark in a user coordinate system of the operating subject, wherein the target operating object is an operating object that needs to be operated among at least one of the operating objects; determine a first position of the target operating object in the tool coordinate system of the robot based on the positions of the target operating object and the reference mark in the user coordinate system, the global image, and the relative physical position of a tool center point of the robot and a camera center point of the camera.

[0007] The operation subject has an operation object, and the operation subject may be a frame in an optical distribution frame (ODF) room, a rack in a server room, a test bench in a laboratory, etc. The specific type of the operation subject is not limited in the embodiment of the present application. When the operation subject is an optical distribution frame, the operation object may be an optical fiber port.

[0008] The reference mark can be a plane mark, and a vertex of the reference mark is used as a reference point, which coincides with the reference point of the operating subject. In order to facilitate identification of the reference mark, the reference mark can be a combination of color and simple shape. For example, the color of the edge of the reference mark is different from the color of the frame of the operating subject and the main part of the reference mark.

[0009] The robot can operate the operation object on the operation body. The robot has a mechanical arm. The camera is fixed on the mechanical arm. The camera can be a monocular camera. A distance measuring device can also be fixed on the robot to measure the distance from the operation body and the operation object. Exemplarily, the distance measuring device can be a laser rangefinder.

[0010] The tool coordinate system is a coordinate system established with the tool center point of the robot as the origin. It is used to describe the position and posture of the tool at the end of the robot. The tool center point (TCP) refers to a point on the tool (e.g., fixture, glue gun, and welding gun) at the end of the robot. The user coordinate system is a concept relative to the tool coordinate system. It is located on the workpiece (e.g., fiber distribution frame) that the robot needs to operate and can also be called the workpiece coordinate system.

[0011] An embodiment of the present application provides a method for determining the position of an operating object. The method arranges reference marks on an operating subject as reference positions, and obtains a global image containing reference marks taken by a camera located on a robot, so that the user coordinate system of the operating subject and the tool coordinate system of the robot can be associated through the camera. Thus, the position of the operating object in the user coordinate system can be converted into the position of the operating object in the tool coordinate system of the robot, which facilitates the robot to subsequently move to the vicinity of the operating object according to the position of the operating object in the tool coordinate system and operate on the operating object.

[0012] In one possible implementation, obtaining a global image of the operating subject captured by a camera located on the robot includes: adjusting the posture of the robot so that the posture angle of the tool coordinate system of the robot is consistent with the posture angle of the user coordinate system of the operating subject; and controlling the camera to capture the global image of the operating subject.

[0013] Among them, the tool coordinate system and the user coordinate system both include the X-axis, Y-axis and Z-axis. When the attitude angles of the tool coordinate system and the user coordinate system are consistent, the X-axis, Y-axis and Z-axis of the tool coordinate system are parallel to the X-axis, Y-axis and Z-axis of the user coordinate system respectively. The X-axis and Y-axis can be parallel to the operation plane where the operation object and the reference mark are located, and the Z-axis is perpendicular to the operation plane. In addition, the optical axis of the camera is parallel to the Z-axis.

[0014] The solution shown in the embodiment of the present application, by capturing a global image when the posture angles of the tool coordinate system and the user coordinate system are consistent, makes it unnecessary to consider the coordinate value of the target operation object on the Z axis of the tool coordinate system (this coordinate value can be directly measured by the ranging device) when determining the position of the target operation object based on the global image. Instead, it is only necessary to calculate the coordinate values ​​of the target operation object on the X axis and Y axis of the tool coordinate system. The number of parameters considered is relatively small. In this way, the position of the target operation object in the tool coordinate system subsequently determined according to the global image is more accurate (the accuracy can reach 0.1mm level), and the algorithm used is also simpler and the processing speed is faster.

[0015] In a possible implementation, determining a first position of the target operation object in the tool coordinate system of the robot based on the positions of the target operation object and the reference mark in the user coordinate system, the global image, and the relative physical position of the tool center point of the robot and the camera center point of the camera includes:

[0016] Based on the positions of the target operation object and the reference mark in the user coordinate system, the global image, and the relative physical positions of the tool center point of the robot and the camera center point of the camera, the position of the target operation object on the X-axis and the Y-axis is determined; and the position of the target operation object on the Z-axis measured by a ranging device located on the robot is obtained.

[0017] The camera may be a monocular camera, and the distance measuring device may be a laser rangefinder.

[0018] The solution shown in the embodiment of the present application is to split the first position of the target operation object in the tool coordinate system of the robot into the position of the X axis, the position of the Y axis and the position of the Z axis in the tool coordinate system. In addition, the position of the target operation object in the X axis and the Y axis is determined through the global image of the operation subject taken by the camera, and the position of the target operation object in the Z axis is determined through the distance measuring device.

[0019] Compared with the position of the target operation object in the X-axis, Y-axis and Z-axis directly determined by the multi-eye structured light camera in the related art. The camera provided by the embodiment of the present application is only used to determine the position of the target operation object in the X-axis and Y-axis, and the position of the target operation object in the Z-axis can be directly measured by the distance measuring device. Like this, the camera provided by the embodiment of the present application can adopt a monocular camera with lower cost, and when determining the position of the target operation object in the X-axis and Y-axis, the algorithm adopted is also relatively simple, and the precision of the position of the target operation object determined in the X-axis and Y-axis is higher. In addition, the position of the target operation object in the Z-axis is directly measured by the distance measuring device, so that the precision of the position of the target operation object determined in the Z-axis is also higher.

[0020] In a possible implementation, controlling the camera to capture a global image of the operating subject includes: acquiring size information of the operating subject; determining a first shooting position based on the size information of the operating subject and parameter information of the camera; controlling the camera to move to the first shooting position, and capturing the global image at the first shooting position.

[0021] The size information of the operating body may include the length and width of the operating surface of the operating body, etc. The parameter information of the camera may include the working distance of the camera.

[0022] The solution shown in the embodiment of the present application determines the first shooting position according to the size information of the operating subject and the parameter information of the camera, so that the captured global image can better present the operating subject, which is beneficial to subsequent calculations based on the global image.

[0023] In a possible implementation, a first information mark is distributed on the same side of the operating body, and the first information mark is a mark storing size information of the operating body. The obtaining of the size information of the operating body includes: identifying the first information mark and obtaining the size information of the operating body.

[0024] The first information mark and the reference mark may be two separate marks, or the first information mark may be integrated into the reference mark. The first information mark may be a mark that can store information, such as a QR code and a barcode.

[0025] The first information mark may also include size information of the operation object.

[0026] The solution shown in the embodiment of the present application stores the size information of the operating subject in the first information flag, so that the method for determining the position of the operating object provided in the embodiment of the present application can be migrated in different scenarios.

[0027] For the robot, there is no need to store any scene information in advance. When the robot migrates to a new scene, it can obtain the size information of the operating subject by scanning the first information mark, and can determine the first shooting position according to the size information of the operating subject and shoot a global image.

[0028] In a possible implementation, acquiring the positions of the target operation object and the reference mark in the user coordinate system of the operation subject includes:

[0029] Acquire the identification of the operating subject; send an operation instruction acquisition request carrying the identification of the operating subject to the host computer; receive the operation instruction sent by the host computer, wherein the operation instruction carries the position of the target operation object and the reference mark in the user coordinate system.

[0030] The operation instruction may also carry information such as the operation type.

[0031] In a possible implementation, a second information mark is distributed on the same side of the operating subject, and the second information mark is a mark storing the identification of the operating subject. The obtaining of the identification of the operating subject includes: identifying the second information mark and obtaining the identification of the operating subject.

[0032] The second information mark and the reference mark may be two separate marks, or the second information mark may be integrated into the reference mark. The second information mark may be a mark that can store information, such as a QR code and a barcode. Exemplarily, the second information mark and the first information mark may be the same mark.

[0033] In a possible implementation, determining a first position of the target operation object in the tool coordinate system of the robot based on the positions of the target operation object and the reference mark in the user coordinate system, the global image, and the relative physical position of the tool center point of the robot and the camera center point of the camera includes:

[0034] Determine the pixel position and pixel size of the reference mark in the global image; determine the mapping ratio of the physical size and the pixel size based on the pixel size of the reference mark and the physical size of the reference mark; determine the first position of the target operation object in the tool coordinate system based on the positions of the target operation object and the reference mark in the user coordinate system, the pixel position of the reference mark in the global image, the mapping ratio, and the relative physical positions of the tool center point and the camera center point.

[0035] The physical size of the reference mark may be a standard value and may be pre-stored in the robot.

[0036] In a possible implementation, determining the first position of the target operation object in the tool coordinate system based on the positions of the target operation object and the reference mark in the user coordinate system, the pixel position of the reference mark in the global image, the mapping ratio, and the relative physical position of the tool center point and the camera center point includes:

[0037] Based on the image center point of the global image, the pixel position of the reference mark in the global image, and the mapping ratio, the relative physical position of the camera center point and the reference mark is determined, wherein the image center point is the mapping point of the camera center point in the global image; based on the relative physical position of the camera center point and the reference mark, and the relative physical position of the tool center point and the camera center point, the relative physical position of the tool center point and the reference mark is determined; based on the relative physical position of the tool center point and the reference mark, and the positions of the target operation object and the reference mark in the user coordinate system, the first position of the target operation object in the tool coordinate system is determined.

[0038] In a possible implementation, determining the first position of the target operation object in the tool coordinate system based on the positions of the target operation object and the reference mark in the user coordinate system, the pixel position of the reference mark in the global image, the mapping ratio, and the relative physical position of the tool center point and the camera center point includes:

[0039] Based on the positions of the target operation object and the reference mark in the user coordinate system, the pixel position of the reference mark in the global image, and the mapping ratio, the pixel position of the target operation object in the global image is determined; based on the image center point of the global image, the pixel position of the target operation object in the global image, and the mapping ratio, the relative physical position of the camera center point and the target operation object is determined, wherein the image center point is the mapping point of the camera center point in the global image; based on the relative physical position of the camera center point and the target operation object, and the relative physical position of the tool center point and the camera center point, the position of the target operation object in the tool coordinate system is determined.

[0040] In a possible implementation manner, after determining the first position of the target operation object in the tool coordinate system of the robot, the method further includes:

[0041] Based on the first position of the target operation object in the tool coordinate system of the robot, determine the second shooting position; control the camera to move to the second shooting position, and shoot a partial image containing the target operation object at the second shooting position; based on the partial image, determine the second position of the target operation object in the tool coordinate system of the robot.

[0042] The solution shown in the embodiment of the present application determines the second position of the target operation object in the tool coordinate system of the robot based on the local image, so that the determined second position of the target operation object in the tool coordinate system is more accurate, so that the method provided in the embodiment of the present application can be applicable to scenarios with higher precision requirements.

[0043] In a possible implementation, determining the second position of the target operation object in the tool coordinate system of the robot based on the local image includes: determining the positions of the target operation object in the X-axis and Y-axis of the tool coordinate system based on the local image; and obtaining the position of the target operation object in the tool coordinate system measured by a ranging device located on the robot.

[0044] The solution shown in the embodiment of the present application is to split the second position of the target operation object in the tool coordinate system of the robot into the position of the X axis, the position of the Y axis and the position of the Z axis in the tool coordinate system. In addition, the position of the target operation object in the X axis and the Y axis is determined through the global image of the operating subject taken by the camera, and the position of the target operation object in the Z axis is determined through the distance measuring device.

[0045] Compared with the position of the target operation object in the X-axis, Y-axis and Z-axis directly determined by the multi-eye structured light camera in the related art. The camera provided by the embodiment of the present application is only used to determine the position of the target operation object in the X-axis and Y-axis, and the position of the target operation object in the Z-axis can be directly measured by the distance measuring device. Like this, the camera provided by the embodiment of the present application can adopt a monocular camera with lower cost, and when determining the position of the target operation object in the X-axis and Y-axis, the algorithm adopted is also relatively simple, and the precision of the position of the target operation object determined in the X-axis and Y-axis is higher. In addition, the position of the target operation object in the Z-axis is directly measured by the distance measuring device, so that the precision of the position of the target operation object determined in the Z-axis is also higher.

[0046] In a possible implementation, determining the second shooting position based on the first position of the target operation object in the tool coordinate system of the robot includes: acquiring size information of the target operation object; and determining the second shooting position based on the size information of the target operation object, parameter information of the camera and the first position.

[0047] The size information of the target operation object may be obtained when scanning the first information mark. The parameter information of the camera includes the working distance of the camera.

[0048] The solution shown in the embodiment of the present application is designed to determine the second shooting position according to the size information of the target operation object, the parameter information of the camera and the first position, so that the target operation object can be better presented in the local image.

[0049] In a possible implementation manner, determining the second position of the target operation object in the tool coordinate system of the robot based on the local image includes:

[0050] Determine the relative pixel position of the center point of the target operation object and the center point of the image in the local image; determine the second position of the target operation object in the tool coordinate system based on the relative pixel position of the center point of the target operation object and the center point of the image, and the mapping ratio of the physical size and pixel size corresponding to the local image.

[0051] In the solution shown in the embodiment of the present application, the second shooting position determined can be the position where the default camera center point and the center point of the target operation object are directly opposite. If the center point of the target operation object and the center point of the image are not coincident in the local image, it means that the first position of the target operation object determined before in the tool coordinate system is not accurate, and the relative physical position of the center point of the target operation object and the center point of the image can be determined based on the relative pixel position of the center point of the target operation object and the center point of the image, and the mapping ratio of the physical size and pixel size corresponding to the local image, and the relative physical position is the error value, and the second position of the target operation object in the current tool coordinate system can be determined based on the error value.

[0052] In a possible implementation, before determining the second position of the target operation object in the tool coordinate system based on the relative pixel position of the center point of the target operation object and the center point of the image, and the mapping ratio between the physical size and the pixel size corresponding to the local image, the method further includes:

[0053] Based on the pixel size of the reference mark in the global image and the physical size of the reference mark, determine the mapping ratio of the physical size and pixel size corresponding to the global image; based on the pixel size of the target operation object in the global image and the mapping ratio of the physical size and pixel size corresponding to the global image, determine the physical size of the target operation object; based on the physical size of the target operation object and the pixel size of the target operation object in the local image, determine the mapping ratio of the physical size and pixel size corresponding to the local image.

[0054] The solution shown in the embodiment of the present application makes the mapping ratio corresponding to the local image more accurate by using the physical size of the target operation object calculated based on the information of the global image instead of the stored theoretical physical size of the target operation object when calculating the mapping ratio corresponding to the local image. In addition, in this case, the theoretical physical size of the target operation object may not be stored in the database.

[0055] Since the manufacturing, installation, and use processes of the operation object, as well as the posture of the target operation object, may cause differences between the theoretical physical size of the operation object and the actual physical size, the mapping ratio directly obtained by using the theoretical physical size of the target operation object and the pixel size of the target operation object may have poor accuracy.

[0056] Since the reference mark can be a plane mark and is relatively simple, the theoretical physical size of the reference mark is closer to the actual physical size. The mapping ratio corresponding to the global image calculated based on the theoretical physical size and pixel size of the reference mark is also relatively accurate. Furthermore, the physical size of the target operation object calculated based on the mapping ratio corresponding to the global image and the pixel size of the target operation object in the global image is also relatively accurate. Therefore, the mapping ratio obtained using the calculated physical size of the target operation object and the pixel size of the target operation object has a higher accuracy.

[0057] In a possible implementation manner, the global image also includes the target operation object, and after acquiring the positions of the target operation object and the reference mark in the user coordinate system of the operation subject, the method further includes:

[0058] Based on the positions of the target operation object and the reference mark in the user coordinate system of the operating subject, and the mapping ratio of the physical size and pixel size corresponding to the global image, the target operation object is determined in the global image; if it is recognized that the target operation object can be operated by the robot, the processing of determining the first position of the target operation object in the tool coordinate system of the robot is performed.

[0059] The solution shown in the embodiment of the present application, after obtaining the positions of the target operation object and the reference mark in the user coordinate system, determines the target operation object in the global image, and performs image recognition on the target operation object to identify whether the target operation object can be operated, so that when it is identified that the target operation object cannot be operated, the determination process of the target operation object position can be terminated in time, because in this case even if the position of the target operation object is determined, the target operation object cannot be operated, thereby saving computer processing resources.

[0060] In a second aspect, a robot is provided, the robot comprising a processor and a memory;

[0061] The memory stores one or more programs, and the one or more programs are configured to be executed by the processor to implement the method as described in any one of the first aspects.

[0062] In a third aspect, a robot system is provided, comprising a robot, a camera and a ranging device, wherein the robot is the robot as described in the second aspect; the camera and the ranging device are fixed on a mechanical arm of the robot.

[0063] The camera may be a monocular camera, and the distance measuring device may be a laser rangefinder.

[0064] According to a fourth aspect, an automation system is provided, comprising an operating body and the robot system according to the third aspect; at least one operating object and a reference mark are distributed on the same side of the operating body.

[0065] Among them, the automation system provided in the embodiment of the present application can be an ODF computer room system, a server computer room system, a laboratory, etc. The embodiment of the present application does not limit the specific type of the automation system.

[0066] In a possible implementation, an information mark is distributed on the same side of the operating body, and the information mark stores one or more of an identification of the operating body, size information of the operating body, and size information of the operating object.

[0067] The information mark and the reference mark may be two separate marks, or the information mark may be integrated into the reference mark.

[0068] In a fifth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes instructions, and when the computer-readable storage medium is run on a robot, the robot executes the method as described in any one of the first aspects.

[0069] In a sixth aspect, a computer program product comprising instructions is provided. When the computer program product is run on a robot, the robot executes the method as described in any one of the first aspects.

[0070] In a seventh aspect, a chip is provided, wherein the chip includes a programmable logic circuit and / or program instructions, and when the chip is running, it is used to implement any method described in the first aspect.

[0071] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0072] An embodiment of the present application provides a method for determining the position of an operating object. The method arranges reference marks on an operating subject as reference positions, and obtains a global image containing reference marks taken by a camera located on a robot, so that the user coordinate system of the operating subject and the tool coordinate system of the robot can be associated through the camera. Thus, the position of the operating object in the user coordinate system can be converted into the position of the operating object in the tool coordinate system of the robot, which facilitates the robot to subsequently move to the vicinity of the operating object according to the position of the operating object in the tool coordinate system and operate on the operating object. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 is a schematic diagram of an operating environment provided by an embodiment of the present application;

[0074] Figure 2 is a schematic diagram of a global image provided by an embodiment of the present application;

[0075] Figure 3 is a flow chart of a method for determining a position of an operation object provided in an embodiment of the present application;

[0076] Figure 4 is a flow chart of a method for determining a position of an operation object provided in an embodiment of the present application;

[0077] Figure 5 is a flow chart of a method for determining a first position of an operation object provided by an embodiment of the present application;

[0078] Figure 6 is a flow chart of a method for determining a first position of an operation object provided by an embodiment of the present application;

[0079] Figure 7 is a partial schematic diagram of a global image provided by an embodiment of the present application;

[0080] Figure 8 It is a schematic diagram of a local image provided in an embodiment of the present application.

[0081] Legend

[0082] 1. Robot, 2. Camera, 3. Operating subject, 31. Operating object, 32. Reference mark, 33. Information mark, 4. Distance measuring device. DETAILED DESCRIPTION

[0083] The existing optical distribution frame (ODF) room has a large inventory and many years of irregular business maintenance, resulting in extremely chaotic fiber layout, difficult room operation and maintenance, difficult problem location, and difficult idle port location. At the same time, in the context of the era of Industry 5.0, the intelligent transformation of the ODF room is imperative. To achieve intelligence, at least the following three points need to be achieved:

[0084] 1. Realize the digitization of the ODF computer room and build a digital mirror of the physical computer room;

[0085] 2. Realize the automation of ODF room operation, and replace manual work with robots for routine operations (such as fiber plugging, unplugging, cleaning and replacement);

[0086] 3. Realize intelligent operation of ODF computer room, automatic fault diagnosis, remote service delivery, and the robot makes appropriate adaptive operations based on environmental perception.

[0087] In the process of realizing automated operations in the ODF computer room, the robot needs to rely on visual guidance to perceive the complex computer room environment, accurately calculate the robot's motion parameters, and determine in real time whether the port is suitable for operation.

[0088] Based on the above requirements, the embodiment of the present application provides a method for determining the position of an operation object, which can be performed by a robot to realize the automated operation of an ODF computer room. It should be noted that the method for determining the position of an operation object provided in the embodiment of the present application can be applied not only in an ODF computer room, but also in any scenario with similar requirements, such as a server room, an operation test bench, etc.

[0089] Below, the hardware involved in the embodiments of the present application is first exemplarily described. Figure 1 As shown, the hardware involved includes a robot system and an operating subject 3.

[0090] The robot system includes a robot 1, a camera 2 and a distance measuring device 4, wherein the camera 2 and the distance measuring device 4 are fixed on the robot arm of the robot 1, and the optical axis of the camera 2, the distance measuring axis of the distance measuring device 4 and a coordinate axis of the tool coordinate system are parallel. The embodiment of the present application does not limit the types of the camera 2 and the distance measuring device 4. For example, the camera 2 can be a monocular camera and the distance measuring device 4 can be a laser rangefinder.

[0091] The operating body 3 has at least one operating object 31 and a reference mark 32, for example, Figure 1As shown, a reference mark 32 and a plurality of identical operation objects 31 are arranged on the operation body 3. The reference mark 32 may be a plane mark, and a vertex of the reference mark 32 is used as a reference point, which coincides with the reference point of the operation body 3 (the reference point may be the coordinate origin of the user coordinate system of the operation body 3). When arranging the reference mark 32, due to errors, the reference point of the reference mark 32 may not completely coincide with the reference point of the operation body 3, so the error between the reference point of the reference mark 32 and the reference point of the operation body 3 is required not to exceed 10% of the size of the operation object 31.

[0092] In order to facilitate identification of the reference mark 32, the reference mark 32 may be a combination of color and simple shape. For example, the color of the edge of the reference mark 32 is different from the color of the frame of the operating body 3 and the main body of the reference mark 32. For example, the color of the edge of the reference mark 32 is red, and the color of the main body of the reference mark 32 is green. An information mark 33 may also be arranged on the operating body 3. The information mark 33 may be a QR code or a barcode, etc. The information mark 33 may store one or more of the identification of the operating body 3, the size information of the operating body 3, and the size information of the operating object 31. The information mark 33 may be integrated into the reference mark 32, or may be arranged separately from the reference mark 32, which is not limited in the embodiment of the present application.

[0093] For ease of understanding, some terms involved in the embodiments of the present application are briefly introduced below:

[0094] Tool center point (TCP): refers to a point on the tool (e.g., fixtures, glue guns, and welding guns) located at the end of the robot.

[0095] Tool coordinate system: refers to a coordinate system established with the tool center point of the robot as the origin, which is used to describe the position and posture of the tool at the end of the robot. Before the robot operates the operation object, it first needs to determine the position of the operation object in the tool coordinate system. The tool coordinate system provided in the embodiment of the present application includes an X-axis, a Y-axis and a Z-axis. The X-axis and the Y-axis are parallel to the operation plane where the operation object 31 and the reference mark 32 are located, and the Z-axis is perpendicular to the operation plane.

[0096] User coordinate system: a concept relative to the tool coordinate system, located on the workpiece (e.g., fiber distribution frame) that the robot needs to operate, and can also be called the workpiece coordinate system. The user coordinate system provided in the embodiment of the present application includes an X-axis, a Y-axis, and a Z-axis, where the X-axis and the Y-axis are parallel to the operating plane, and the Z-axis is perpendicular to the operating plane.

[0097] Figure 31 is a flow chart of a method for determining the position of an operation object shown in an embodiment of the present application. The method can be applied to the robot 1, and includes the following steps:

[0098] Step 301 : obtaining a global image of the operating subject 3 captured by the camera 2 located on the robot 1 .

[0099] The operating body 3 is a body where the operating object 31 of the robot 1 is located, a reference mark 32 and at least one operating object 31 are distributed on the same side of the operating body 3 , and the global image is an image including the reference mark 32 .

[0100] Step 302 : obtaining the positions of the target operation object and the reference mark 32 in the user coordinate system of the operation subject 3 .

[0101] The target operation object is the operation object 31 that needs to be operated on among the at least one operation object 31 .

[0102] Step 303, based on the positions of the target operation object and the reference mark 32 in the user coordinate system, the global image, and the relative physical positions of the tool center point of the robot 1 and the camera center point of the camera 2, determine the first position of the target operation object in the tool coordinate system of the robot 1.

[0103] Figure 4 FIG. 1 is a flow chart of another method for determining the position of an operation object shown in an embodiment of the present application. The method can be applied to the robot 1 described above. The processing flow of the method will be described in more detail below in conjunction with a specific implementation manner, including the following steps:

[0104] Step 401 : obtaining a global image of the operating subject 3 captured by the camera 2 located on the robot 1 .

[0105] The operating body 3 is a body where the operating object 31 of the robot 1 is located, a reference mark 32 and at least one operating object 31 are distributed on the same side of the operating body 3 , and the global image is an image including the reference mark 32 .

[0106] When shooting a global image, the posture of the robot 1 can be adjusted first so that the posture angle of the tool coordinate system of the robot 1 is consistent with the posture angle of the user coordinate system of the operating subject 3, and the global image is shot when the optical axis of the camera 2 is perpendicular to the operating subject 3. When the posture angles of the tool coordinate system and the user coordinate system are consistent, the X-axis, Y-axis and Z-axis of the user coordinate system are respectively parallel to the X-axis, Y-axis and Z-axis of the tool coordinate system, and the optical axis of the camera 2 is parallel to the Z-axis.

[0107] Assume that the operation plane of the operating subject 3 is the plane where the X-axis and the Y-axis are located, and the Z-axis is perpendicular to the operation plane. By shooting a global image when the posture angles of the tool coordinate system and the user coordinate system are consistent, when determining the position of the target operation object based on the global image, it is not necessary to consider the coordinate value of the Z-axis of the target operation object in the tool coordinate system (this coordinate value can be directly measured by the distance measuring device), but only the coordinate values ​​of the X-axis and Y-axis of the target operation object in the tool coordinate system need to be calculated, and the number of parameters considered is small. In this way, the position of the target operation object in the tool coordinate system determined later based on the global image is more accurate (the accuracy can reach 0.1mm level), and the algorithm used is also simpler, and the processing speed is also faster.

[0108] The method of adjusting the posture of the robot 1 is not specifically limited in the present embodiment. For example, the distance measuring device 4 (laser distance meter) can be used to adjust the posture of the robot 1. The specific process of posture adjustment can be described as follows:

[0109] First, when the robot arm moves, the laser return distance value is obtained in real time, and the movement is stopped when the specified distance (that is, the distance of the operating body 3) is reached.

[0110] Then, three depth values ​​(distance values ​​from the operating body 3) are obtained by scanning on the operating body 3, and the rotation angle of the robot tool center point coordinate system around the base coordinate is calculated to correct the posture.

[0111] In addition, in addition to adjusting the posture of the robot 1, in order to make the operating subject 3 better presented in the global image, a first shooting position can be determined, and the global image can be shot at the first shooting position. The first shooting position includes a first shooting distance, which is the distance from the operating subject 3 when the camera 2 shoots the global image.

[0112] The embodiment of the present application does not limit the method for determining the first shooting distance. For example, the first shooting distance can ensure that the operating subject 3 is fully presented in the global image at a sufficiently large scale, and the first shooting distance can be calculated based on the size information of the operating subject 3 and the parameter information of the camera.

[0113] In a specific implementation, the first shooting distance may be a predetermined value. It may be an empirical value or a value pre-calculated based on the size information of the operating subject 3. For example, when the robot 1 is used in certain fixed scenes, the size information of the operating subject 3 and the operating object 31 has been determined, and the first shooting distance may be directly calculated based on the size information and the working parameter information of the camera 2. Then, each time the robot 1 captures a global image, the global image may be captured at the first shooting distance.

[0114] In another specific implementation, before taking a global image each time, the robot 1 may first obtain the size information of the operating subject 3. Then, based on the size information of the operating subject 3 and the parameter information of the camera 2, the first shooting position is determined. The camera 2 is controlled to move to the first shooting position, and the global image is taken at the first shooting position.

[0115] In addition, the first shooting position may also include the coordinate values ​​of the camera 2 on the X-axis and the Y-axis, and the coordinate values ​​of the camera 2 on the X-axis and the Y-axis may be determined when the image recognition of the camera 2 shows that the operating subject 3 is completely presented in the global image. It is understandable that since the coordinate values ​​of the camera 2 on the X-axis and the Y-axis are not values ​​that require very high precision, many methods can be used to ensure that the operating subject 3 is completely presented in the global image, and the embodiments of the present application will not be repeated in detail.

[0116] The embodiment of the present application does not specifically limit the method for acquiring the size information of the operating body 3.

[0117] In a specific implementation, a first information mark is also distributed on the same side of the operating body 3, and the first information mark is a mark storing the size information of the operating body 3, and the global image also includes the first information mark. Then the robot 1 can identify the first information mark and obtain the size information of the operating body 3 stored in the first information mark. The first information mark and the reference mark 32 can be distributed on the operating body 3 respectively, and the first information mark can also be integrated in the reference mark 32, which is not specifically limited in the embodiment of the present application.

[0118] Step 402 : obtaining the positions of the target operation object and the reference mark 32 in the user coordinate system of the operation subject 3 .

[0119] The target operation object is the operation object 31 that needs to be operated on among the at least one operation object 31 .

[0120] The positions of the target operation object and the reference mark 32 in the user coordinate system of the operating subject 3 can also be understood as the relative positions of the target operation object and the reference mark 32 .

[0121] In some cases, the position of the reference mark 32 is determined as the reference position (such as the coordinate origin of the user coordinate system), and then the position of the target operation object in the user coordinate system is obtained, which should be considered as the position of the target operation object and the reference mark 32 in the user coordinate system. It can be understood that the position of the target operation object and the reference mark 32 in the user coordinate system of the operating subject 3 is a physical position.

[0122] The present application does not limit the method for obtaining the position of the target operation object and the reference mark 32 in the user coordinate system. In a specific implementation, the position of the target operation object and the reference mark 32 in the user coordinate system can be obtained from the host computer.

[0123] Exemplarily, the process of obtaining the position of the target operation object and the reference mark 32 in the user coordinate system from the host computer can be described as follows: the robot 1 first obtains the identification of the operation subject 3, and then sends an operation instruction acquisition request carrying the identification of the operation subject 3 to the host computer. After receiving the operation instruction acquisition request, the host computer sends an operation instruction to the robot 1 based on the identification of the operation subject 3, and the operation instruction carries the position of the target operation object and the reference mark 32 in the user coordinate system. After that, the robot receives the operation instruction and can obtain the position of the reference mark 32 and the target operation object in the user coordinate system.

[0124] It should be noted that the determination of the target operation object and the type of operation to be performed on the target operation object can also be determined when the position of the target operation object and the reference mark 32 in the user coordinate system is obtained. Before the position of the target operation object and the reference mark 32 in the user coordinate system is obtained, the robot 1 does not determine the target operation object, nor the type of operation to be performed on the target operation object. All of this information can be carried in the operation instructions of the host computer.

[0125] The embodiment of the present application does not limit the method for obtaining the identification of the operating subject 3.

[0126] In a specific implementation, a second information mark is also distributed on the same side of the operating body 3, and the second information mark is a mark storing the identification of the operating body 3. Then the robot 1 can obtain the identification of the operating body 3 by identifying the second information mark. The identification of the second information mark can be performed in the global image, or before taking the global image, during the scanning process of the camera 2. The second information mark can be the same information mark as the first information mark (such as Figure 2 The information mark 33) in the figure can also be a separate mark from the second information mark.

[0127] Of course, the above methods for obtaining the positions of the target operation object and the reference mark 32 in the user coordinate system, and the method for obtaining the identification of the operation subject 3, are merely exemplary and do not constitute a limitation on the embodiments of the present application. In practical applications, other methods may also be used to obtain the above contents.

[0128] Step 403, based on the positions of the target operation object and the reference mark 32 in the user coordinate system of the operating subject 3, and the mapping ratio of the physical size and pixel size corresponding to the global image, determine the target operation object in the global image, and identify whether the target operation object can be operated by the robot 1.

[0129] The global image is an image including the target operation object and the reference mark 32. The mapping ratio of the physical size and the pixel size corresponding to the global image can be calculated by the physical size and the pixel size of the reference mark 32.

[0130] After obtaining the positions of the target operation object and the reference mark 32 in the user coordinate system and capturing the global image, the relative pixel positions of the target operation object and the reference mark 32 in the global image can be calculated based on the relative physical positions of the target operation object and the reference mark 32 and the mapping ratio.

[0131] Then, according to the pixel position of the reference mark 32 in the global image and the relative pixel position, the operation object 31 satisfying the relative pixel position is found in the global image, and the operation object 31 is determined as the target operation object.

[0132] Afterwards, an image recognition algorithm is used to identify whether the target operation object can be operated by the robot, for example, whether it is blocked or damaged. If it is identified that the target operation object can be operated by the robot 1, the process goes to step 404; otherwise, no subsequent processing is performed to save computer resources, because in this case, even if the position of the target operation object in the tool coordinate system is determined, subsequent operations cannot be performed.

[0133] Step 404, based on the positions of the target operation object and the reference mark 32 in the user coordinate system, the global image, and the relative physical positions of the tool center point of the robot 1 and the camera center point of the camera 2, determine the first position of the target operation object in the tool coordinate system of the robot 1.

[0134] The relative physical position between the tool center point of the robot 1 and the camera center point of the camera 2 can be determined after the camera 2 is installed, and this relative physical position can be stored in the robot 1 .

[0135] The process of determining the first position of the target operation object in the tool coordinate system of the robot 1 can be described as follows:

[0136] First, the pixel position and pixel size of the reference marker 32 are determined in the global image.

[0137] Then, based on the pixel size of the reference mark 32 and the physical size of the reference mark 32 , a mapping ratio of the physical size and the pixel size is determined.

[0138] Finally, based on the positions of the target operation object and the reference mark 32 in the user coordinate system, the pixel position of the reference mark 32 in the global image, the mapping ratio, and the relative physical positions of the tool center point and the camera center point, the first position of the target operation object in the tool coordinate system is determined.

[0139] In order to better understand the present application, the process of determining the first position of the target operation object in the tool coordinate system is described in more detail below:

[0140] like Figure 5 As shown, a possible implementation method for determining the first position of the target operation object in the tool coordinate system is provided.

[0141] Step 404a, based on the image center point of the global image, the pixel position of the reference mark 32 in the global image, and the mapping ratio, determine the relative physical position of the camera center point and the reference mark 32.

[0142] The image center point is the mapping point of the camera center point in the global image.

[0143] In implementation, based on the image center point of the global image and the pixel position of the reference mark 32 in the global image, the relative pixel position of the image center point and the reference mark 32 can be determined. Then, according to the mapping ratio, the relative pixel position is mapped to the relative physical position, that is, the relative physical position of the camera center point and the reference mark 32 is obtained.

[0144] Step 404b, based on the relative physical position of the camera center point and the reference mark 32, and the relative physical position of the tool center point and the camera center point, determine the relative physical position of the tool center point and the reference mark 32.

[0145] Step 404c, based on the relative physical positions of the tool center point and the reference mark 32, and the positions of the target operation object and the reference mark 32 in the user coordinate system, determine the first position of the target operation object in the tool coordinate system.

[0146] like Figure 6 As shown, another possible implementation method for determining the first position of the target operation object in the tool coordinate system is provided.

[0147] Step 404A, based on the positions of the target operation object and the reference mark 32 in the user coordinate system, the pixel position of the reference mark 32 in the global image, and the mapping ratio, determine the pixel position of the target operation object in the global image.

[0148] Step 404B, based on the image center point of the global image, the pixel position of the target operation object in the global image, and the mapping ratio, determine the relative physical position of the camera center point and the target operation object.

[0149] The image center point is the mapping point of the camera center point in the global image.

[0150] Step 404C, based on the relative physical positions of the camera center point and the target operation object, and the relative physical positions of the tool center point and the camera center point, determine the first position of the target operation object in the tool coordinate system.

[0151] It should be noted that the above two implementation methods are not all implementation methods. Those skilled in the art can understand that other implementation methods can also be used to determine the first position of the target operation object in the tool coordinate system based on the position of the target operation object and the reference mark 32 in the user coordinate system, the pixel position of the reference mark 32 in the global image, the mapping ratio, and the relative physical position of the tool center point and the camera center point.

[0152] It should also be added that the first position includes the position of the target operation object on the X-axis, Y-axis and Z-axis. Based on the position of the target operation object and the reference mark in the user coordinate system, the global image, and the relative physical position of the tool center point of the robot and the camera center point of the camera, the position of the target operation object on the X-axis and Y-axis can be determined.

[0153] The position of the target operation object on the Z axis of the tool coordinate system can be measured by the distance measuring device 4 .

[0154] Exemplarily, in a specific implementation, when the distance measuring device 4 measures the position of the target operation object in the Z axis, the robot 1 drives the distance measuring device 4 to first move to a position relative to the target operation object, and then directly measures the distance from the target operation object (i.e., the position of the target operation object in the Z axis). In another specific implementation, the distance measuring device 4 can also measure the distance from the operation plane of the operating subject 3, and then, based on the pre-stored distance of the target operation object relative to the operation plane (for example, the target operation object can protrude a certain length relative to the operation plane), determine the position of the target operation object in the Z axis.

[0155] After determining the first position of the target operation object in the tool coordinate system, in a specific implementation, the working position can be determined, and the robot 1 can be controlled to move to the working position and operate the target operation object at the working position.

[0156] In another specific implementation, in some scenarios with very high precision requirements, for example, scenarios with a precision requirement of 0.1 mm, the error of the determined first position may be large, and the process may proceed to execute steps 405 and 406 to determine a more precise position of the target operation object in the tool coordinate system.

[0157] Step 405 , based on the first position of the target operation object in the tool coordinate system of the robot 1 , determine a second shooting position, control the camera 2 to move to the second shooting position, and shoot a partial image containing the target operation object at the second shooting position.

[0158] In the solution shown in the embodiment of the present application, the coordinates of the center point of the camera on the X-axis and Y-axis at the second shooting position can be the same as the coordinates of the center point of the target operation object on the X-axis and Y-axis, wherein the X-axis and the Y-axis are parallel to the operation plane of the operation subject 3, and the Z-axis is perpendicular to the operation plane of the operation subject 3. That is, it is ensured as much as possible that the camera 2 is facing the target operation object to shoot a partial image.

[0159] It should be noted that the fact that the coordinates of the camera center point on the X-axis and Y-axis at the second shooting position determined in step 405 are the same as the coordinates of the center point of the target operation object on the X-axis and Y-axis does not mean that the actual coordinate values ​​of the camera center point and the center point of the target operation object on the X-axis and Y-axis are the same, because the determined coordinates of the center point of the target operation object on the X-axis and Y-axis may have errors and are not true values.

[0160] In addition, the determined second shooting position includes not only the coordinates on the X-axis and the Y-axis, but also the coordinate value on the Z-axis (or referred to as the second shooting distance). Exemplarily, the second shooting distance should be able to ensure that the target operation object is presented in the partial image with a sufficiently large proportion, and the second shooting distance can be calculated based on the size information of the target operation object and the parameter information of the camera 2.

[0161] In a specific implementation, the second shooting distance may be an empirical value or a pre-calculated value. When determining the second shooting position, the pre-stored second shooting distance may be obtained.

[0162] In another specific implementation, the second shooting distance may also be calculated in real time, and the processing process may be as follows: obtaining the size information of the target operation object, and determining the second shooting position based on the size information of the target operation object, the parameter information of the camera 2 and the first position.

[0163] The size information of the target operation object may be sent by the host computer or stored in the information flag, which is not limited in the embodiment of the present application. When the size information of the target operation object is stored in the information flag, the information flag may be the same as the first information flag and the second information flag, and may be integrated in the reference flag 32.

[0164] Step 406 : determining a second position of the target operation object in the tool coordinate system of the robot 1 based on the local image.

[0165] The processing process of step 406 is described by taking the X-axis and Y-axis coordinate values ​​of the camera center point at the second shooting position being the same as the X-axis and Y-axis coordinate values ​​of the center point of the target operation object (determined in step 405) as an example.

[0166] First, the relative pixel positions of the center point of the target operation object and the center point of the image are determined in the local image.

[0167] Then, based on the relative pixel positions of the center point of the target operation object and the center point of the image, and the mapping ratio of the physical size and the pixel size corresponding to the local image, the second position of the target operation object in the tool coordinate system is determined.

[0168] If the relative pixel positions of the target operation object and the center point of the image are the same, it means that the center point of the camera is facing the center point of the target operation object, and there is no error in the first position determined. If the relative pixel positions of the target operation object and the center point of the image are different, it means that the center point of the camera deviates from the target operation object, and there is an error in the first position determined before, and the relative physical position of the center point of the target operation object and the center point of the image is the error value.

[0169] Exemplarily, the relative physical position of the center point of the target operation object and the center point of the camera can be calculated by the following formula:

[0170]

[0171] Among them, Figure 8 As shown, is the pixel position of the center point of the target operation object in the local image, O(x o ,y o ) is the image center point of the local image, dx and dy are the determined relative physical positions, and It is the mapping ratio between the physical size and pixel size of the local image.

[0172] The present application does not limit the method for determining the mapping ratio of the physical size and pixel size corresponding to the local image. In a specific implementation, the process of determining the mapping ratio can be described as follows:

[0173] First, based on the pixel size of the reference mark 32 in the global image and the physical size of the reference mark 32 , a mapping ratio between the physical size and the pixel size corresponding to the global image is determined.

[0174] Exemplarily, the mapping ratio corresponding to the global image may be determined according to the following formula:

[0175]

[0176] Among them, X o and Y o is the known physical size of the reference mark 32, such as Figure 7 As shown, a(x a ,y a )、b(x b ,y b )、c(x c ,y c ) is the pixel coordinate of the reference marker 32 in the global image, D x and D y It is the mapping ratio between the physical size and pixel size corresponding to the global image.

[0177] Then, based on the pixel size of the target operation object in the global image and the mapping ratio between the physical size and the pixel size corresponding to the global image, the physical size of the target operation object is determined.

[0178]

[0179] Among them, X p and Y p It is the calculated physical size of the target operation object. There is a difference between the calculated physical size and the theoretical physical size stored in the database. are the pixel coordinates of the target operation object in the global image.

[0180] Finally, based on the physical size of the target operation object and the pixel size of the target operation object in the local image, a mapping ratio between the physical size and the pixel size corresponding to the local image is determined.

[0181]

[0182] in, and It is the mapping ratio between the physical size and pixel size of the local image. It is the pixel coordinate of the target operation object in the local image.

[0183] In another possible implementation, a pre-stored theoretical physical size of the target operation object and a pixel size of the target operation object may be divided to obtain a mapping ratio between the physical size and the pixel size corresponding to the local image.

[0184] It should be noted that the second position includes the position of the target operation object on the X-axis, Y-axis and Z-axis. The position of the target operation object on the X-axis and Y-axis can be determined based on the position of the target operation object and the reference mark in the user coordinate system, the global image, and the relative physical position of the tool center point of the robot and the camera center point of the camera.

[0185] The position of the target operation object on the Z axis of the tool coordinate system can be measured by the distance measuring device 4 .

[0186] Exemplarily, in a specific implementation method, when the distance measuring device 4 measures the position of the target operation object on the Z axis, the robot 1 drives the distance measuring device 4 to move to a position relative to the target operation object first, and then directly measures the distance to the target operation object (that is, the position of the target operation object on the Z axis).

[0187] It should also be noted that steps 403, 405 and 406 are optional steps. Those skilled in the art can understand that when the first position of the target operation object in the tool coordinate system meets the accuracy requirements, it is not necessary to perform the subsequent steps 405 and 406. In addition, in some scenarios, it is not necessary to use the processing of step 403 to identify whether the target operation object can be operated by the robot. It should also be noted that the order of steps 401 and 402 can be interchanged.

[0188] In summary, the method for determining the position of an operation object provided in the embodiment of the present application has at least the following beneficial effects:

[0189] The relative position of the tool center point to the reference mark is obtained through vision, which is highly efficient. Using a high-definition monocular camera, under certain posture conditions, μm-level guidance accuracy can be obtained. The price of a monocular camera is cheap and the corresponding algorithm is simple. The target operation object is partially photographed, and the precise position of the target operation object is calculated based on the local image to improve the operation accuracy. The method of local photography plus prior knowledge is less complex than the global photography algorithm.

[0190] Below, the embodiments of the present application are described in conjunction with specific scenarios.

[0191] The method for determining the position of the operation object provided in the embodiment of the present application can be applied in the operator's transmission network and access network room. For various frames in the operator's transmission network or access network room, including the Lucent connector or local connector (LC) head oblique insertion frame, the square connector (SC) head straight insertion frame, and the SC head oblique insertion frame, as long as there is a plane (for example, a 35mmx35mm plane) on which the reference mark 31 can be arranged on the frame operation plane or on the parallel operation plane adjacent to it (such as a rack), and the frame has prior knowledge of the template, the method provided in the embodiment of the present application can accurately guide the movable operation equipment such as the robot arm to the target position with an accuracy of μm level.

[0192] The following is a description of the method for determining the position of an operation object provided in an embodiment of the present application:

[0193] (1) The robot 1 is powered on and drives the camera 2 to move to the vicinity of the reference mark 32 of the machine frame, and stops moving within a range of 200 mm to 300 mm from the machine frame. The camera 2 captures a global image.

[0194] (2) by identifying the reference mark 32, obtaining the frame number, the corresponding physical distance between adjacent pixels, and the pixel position of the reference mark 32;

[0195] (3) Use the chassis number to interact with the host computer to obtain the relative physical position of the port to be operated and the reference mark 32, and at the same time, obtain the specific operation type, such as optical fiber plugging and unplugging, port detection, and port cleaning operations;

[0196] (4) Based on image recognition, determine whether the port is blocked. If it is blocked, end the program; if it is not blocked, proceed to the next step;

[0197] (5) determining a first position according to the relative physical position of the port and the reference mark 32, the position difference between the camera center point and the reference mark 32, and the relative position between the camera center point and the tool center point, and moving to the vicinity of the port to be operated according to the first position;

[0198] (6) Take a local image (taken within a range of 180 mm to 220 mm from the port) to determine the second position of the port in the tool coordinate system;

[0199] (7) The distance measurement device 4 completes the distance measurement between the port and the center point of the tool, and finally the robot 1 moves to the target working position to complete the operation.

[0200] In addition, the method for determining the position of the operation object provided in the embodiment of the present application can also be used for guided positioning of high-precision automated operations on a laboratory bench, including adding reagents to a test kit one by one, replacing reagent tubes, and plugging and unplugging ports of laboratory instruments.

[0201] A rectangular operation space for various experimental operations is planned on the experimental table, and a reference mark 32 is arranged at a marked position inside the operation space. The QR code inside the reference mark 32 stores the operation space number, operation space scale, position coordinates of the reference mark 32 in the operation space, scale of the reference mark 32, scale of operation objects such as reagent kits, and prior knowledge of the template of the operation object.

[0202] The following is a description of the method for determining the position of an operation object provided in an embodiment of the present application:

[0203] (1) The robot 1 is powered on and moves to the vicinity of the reference mark 32. The shooting distance is adjusted to identify the internal information mark of the reference mark 32, obtain the scale of the operation space, the scale of the reference mark and the scale of the operation object, calculate the optimal shooting distance according to these three scales and move, so that the operation subject appears in the captured image in a suitable proportion. At the same time, the camera focal length is set according to the shooting distance to ensure the capture of high-definition images;

[0204] (2) obtaining the pixel positions corresponding to the prior knowledge of the operation object template and the reference mark 32, and calculating the corresponding physical distances between adjacent pixels;

[0205] (3) Use the operation space number to interact with the host computer to obtain the coordinate value of the operation object to be operated from the reference mark 32, and at the same time obtain the specific instructions corresponding to the operation object;

[0206] (4) The robot 1 moves to the vicinity of the operation object according to the operation object corresponding to the operation instruction, the position difference between the camera center point and the reference mark 32, and the position between the camera center point and the tool center point, calculates the optimal shooting distance of the port front end in combination with the scale of the operation object, and moves to the distance to shoot the port;

[0207] (5) performing precision interpolation on the first position of the operation object according to the captured partial image;

[0208] (6) Finally, robot 1 moves to the working position, completes the operation, and returns to the initial position.

[0209] The embodiment of the present application also provides a robot, the robot comprising a processor and a memory. The memory stores one or more programs, the one or more programs are configured to be executed by the processor to implement the method for determining the position of an operation object provided in the embodiment of the present application.

[0210] The present application also provides a robot system, such as Figure 1 As shown, the robot system includes a robot 1, a camera 2 and a distance measuring device 4. The camera 2 and the distance measuring device 4 are fixed on the robot arm of the robot 1.

[0211] The present application also provides an automated system. Figure 1 As shown, the automation system comprises an operating body 3 and the above-mentioned robot system. At least one operating object 31 and a reference mark 32 are distributed on the same side of the operating body 3.

[0212] In a specific implementation, an information mark is distributed on the same side of the operating body 3 , and the information mark stores one or more of the identification of the operating body 3 , the size information of the operating body 3 , and the size information of the operating object 31 .

[0213] An embodiment of the present application also provides a computer-readable storage medium, which includes instructions. When the computer-readable storage medium runs on a robot, the robot executes the method for determining the position of an operation object provided in an embodiment of the present application.

[0214] The embodiment of the present application also provides a computer program product containing instructions. When the computer program product runs on a robot, the robot executes the method for determining the position of an operation object provided in the embodiment of the present application.

[0215] An embodiment of the present application also provides a chip, which includes a programmable logic circuit and / or program instructions, and when the chip is running, it is used to implement the method for determining the position of an operation object provided in an embodiment of the present application.

[0216] In the embodiments of the present application, the terms “first” and “second” are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0217] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining the position of an operation object, It is characterized in that The method is applied in a robot (1), and comprises: Acquire a global image of an operating subject (3) captured by a camera (2) located on the robot (1), wherein the operating subject (3) is a subject where an operating object (31) of the robot (1) is located, an operating plane of the operating subject (3) is distributed with reference marks (32) and at least one of the operating objects (31), the global image is an image including the reference marks (32), the global image is captured when the tool coordinate system of the robot (1) and the user coordinate system of the operating subject (3) have the same attitude angle, the user coordinate system is located on the operating subject (3), the tool coordinate system refers to a coordinate system established with the tool center point of the robot (1) as the origin, the X-axis and Y-axis of the user coordinate system are parallel to the operating plane, and the Z-axis is perpendicular to the operating plane; Acquiring the positions of a target operation object and the reference mark (32) in the user coordinate system, wherein the target operation object is an operation object (31) that needs to be operated among at least one of the operation objects (31); determining the pixel position and pixel size of the reference marker (32) in the global image; Determining a mapping ratio between the physical size and the pixel size based on the pixel size of the reference mark (32) and the physical size of the reference mark (32); Based on the positions of the target operation object and the reference mark (32) in the user coordinate system, the pixel position of the reference mark (32) in the global image, the mapping ratio, and the relative physical position of the tool center point and the camera center point of the camera (2), the first position of the target operation object in the tool coordinate system is determined, wherein the first position includes the position of the target operation object on the X-axis and the Y-axis.

2. The method according to claim 1, It is characterized in that The step of acquiring a global image of an operating subject (3) photographed by a camera (2) located on the robot (1) comprises: Adjusting the posture of the robot (1) so that the posture angle of the tool coordinate system of the robot (1) is consistent with the posture angle of the user coordinate system of the operating subject (3); The camera (2) is controlled to capture a global image of the operating subject (3).

3. The method according to claim 2, It is characterized in that The method further comprises: The position of the target operation object on the Z axis measured by a distance measuring device (4) located on the robot (1) is obtained.

4. The method according to claim 2, It is characterized in that The controlling the camera (2) to capture a global image of the operating subject (3) comprises: Acquiring size information of the operating subject (3); Determining a first shooting position based on the size information of the operating subject (3) and the parameter information of the camera (2); The camera (2) is controlled to move to the first shooting position, and the global image is shot at the first shooting position.

5. The method according to claim 4, It is characterized in that The operating plane of the operating body (3) is also distributed with a first information mark, the first information mark being a mark storing the size information of the operating body (3), and the step of obtaining the size information of the operating body (3) comprises: The first information mark is identified to obtain the size information of the operating subject (3).

6. The method according to claim 1, It is characterized in that The step of acquiring the positions of the target operation object and the reference mark (32) in the user coordinate system of the operation subject (3) comprises: Obtaining the identification of the operating subject (3); Sending an operation instruction acquisition request carrying the identification of the operation subject (3) to the upper computer; An operation instruction sent by the host computer is received, wherein the operation instruction carries the position of the target operation object and the reference mark (32) in the user coordinate system.

7. The method according to claim 6, It is characterized in that The operating plane of the operating subject (3) is also distributed with a second information mark, the second information mark being a mark storing the identification of the operating subject (3), and the obtaining of the identification of the operating subject (3) comprises: The second information mark is identified to obtain the identification of the operating subject (3).

8. The method according to claim 1, It is characterized in that The method of determining a first position of the target operation object in the tool coordinate system based on the positions of the target operation object and the reference mark (32) in the user coordinate system, the pixel position of the reference mark (32) in the global image, the mapping ratio, and the relative physical position of the tool center point and the camera center point of the camera (2) comprises: Determining the relative physical position of the camera center point and the reference mark (32) based on the image center point of the global image, the pixel position of the reference mark (32) in the global image, and the mapping ratio, wherein the image center point is the mapping point of the camera center point in the global image; Determining the relative physical position of the tool center point and the reference mark (32) based on the relative physical position of the camera center point and the reference mark (32), and the relative physical position of the tool center point and the camera center point; Based on the relative physical position of the tool center point and the reference mark (32), and the positions of the target operation object and the reference mark (32) in the user coordinate system, a first position of the target operation object in the tool coordinate system is determined.

9. The method according to any one of claims 1 to 8, It is characterized in that After determining the first position of the target operation object in the tool coordinate system, the method further includes: Determining a second shooting position based on a first position of the target operation object in the tool coordinate system; Controlling the camera (2) to move to the second shooting position, and shooting a partial image containing the target operation object at the second shooting position; Based on the local image, a second position of the target operation object in the tool coordinate system is determined.

10. The method according to claim 9, It is characterized in that The determining, based on the local image, a second position of the target operation object in the tool coordinate system comprises: Based on the local image, determining the position of the target operation object on the X-axis and the Y-axis of the tool coordinate system; The position of the target operation object on the Z axis of the tool coordinate system measured by a distance measuring device (4) located on the robot (1) is obtained.

11. The method according to claim 9, It is characterized in that The determining of the second shooting position based on the first position of the target operation object in the tool coordinate system includes: Obtaining size information of the target operation object; The second shooting position is determined based on the size information of the target operation object, the parameter information of the camera (2) and the first position.

12. The method according to claim 9, It is characterized in that The determining, based on the local image, a second position of the target operation object in the tool coordinate system comprises: Determining the relative pixel position of the center point of the target operation object and the center point of the image in the partial image; Based on the relative pixel positions of the center point of the target operation object and the center point of the image, and the mapping ratio of the physical size and the pixel size corresponding to the local image, the second position of the target operation object in the tool coordinate system is determined.

13. The method according to claim 12, It is characterized in that The global image also includes the target operation object. Before determining the second position of the target operation object in the tool coordinate system based on the relative pixel position of the center point of the target operation object and the center point of the image, and the mapping ratio of the physical size and the pixel size corresponding to the local image, the method further includes: Determine a mapping ratio between the physical size and the pixel size corresponding to the global image based on the pixel size of the reference mark (32) in the global image and the physical size of the reference mark (32); Determine the physical size of the target operation object based on the pixel size of the target operation object in the global image and the mapping ratio between the physical size and the pixel size corresponding to the global image; Based on the physical size of the target operation object and the pixel size of the target operation object in the partial image, a mapping ratio between the physical size and the pixel size corresponding to the partial image is determined.

14. The method according to any one of claims 1 to 8, It is characterized in that The global image also includes the target operation object. After acquiring the positions of the target operation object and the reference mark (32) in the user coordinate system, the method further includes: Based on the positions of the target operation object and the reference mark (32) in the user coordinate system of the operation subject (3), and the mapping ratio between the physical size and the pixel size corresponding to the global image, the target operation object is determined in the global image; If it is recognized that the target operation object can be operated by the robot (1), the process of determining the first position of the target operation object in the tool coordinate system of the robot (1) is performed.

15. A robot, It is characterized in that The robot comprises a processor and a memory; The memory stores one or more programs, and the one or more programs are configured to be executed by the processor to implement the method according to any one of claims 1 to 14.

16. A robot system, It is characterized in that The robot system comprises a robot (1), a camera (2) and a distance measuring device (4), wherein the robot (1) is the robot according to claim 15; The camera (2) and the distance measuring device (4) are fixed on the mechanical arm of the robot (1).

17. An automated system, It is characterized in that The automation system comprises an operating subject (3) and a robot system as claimed in claim 16; An operating plane of the operating body (3) is distributed with at least one operating object (31) and a reference mark (32).

18. The automated system according to claim 17, It is characterized in that The operating plane of the operating body (3) is also distributed with information marks, and the information marks store one or more of the identification of the operating body (3), the size information of the operating body (3), and the size information of the operating object (31).

Images