Target positioning method and device

Abstract

The embodiment of the invention provides a target positioning method and device, and relates to the technical field of robots, and the method comprises the steps: recognizing an image processing operator used for achieving the image processing operation represented by each first image operation node; loading the identified image processing operator from a pre-packaged image processing operator, and instantiating the loaded image processing operator to obtain an instantiated operator; determining a calling sequence of the instantiation operators according to pointer directions among the first image operation nodes; according to the calling sequence, calling each instantiation operator processing operator input image in sequence, and obtaining a first position of a task target in a to-be-processed image collected by a camera under an image coordinate system; and based on the first position and the calibration relation, the second position of the task target in the mechanical arm base coordinate system is determined. By applying the scheme provided by the embodiment of the invention, the task target of the mechanical arm can be positioned.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a target positioning method and device. Background Art

[0002] With the continuous development of industrial automation and intelligent manufacturing technology, robotic arm control technology has been widely used. Its core lies in using robotic arms to perform various operations according to task objectives and complete set tasks.

[0003] For example, in industrial production line scenarios, robotic arms are often installed alongside the line to perform tasks such as assembly and assembling workpieces and components. These tasks can include grasping and placing the workpieces at designated locations.

[0004] Before the robot arm can perform the set operation on the task target, it needs to locate the task target first. Therefore, target positioning is crucial for the robot arm to perform the task.

[0005] In view of the above situation, an embodiment of the present application provides a target positioning method to locate the task target of a robotic arm. Summary of the Invention

[0006] The purpose of the embodiments of the present application is to provide a target positioning method and device to locate the task target of a robotic arm.

[0007] The specific technical solutions are as follows:

[0008] In a first aspect, an embodiment of the present application provides a target positioning method, the method comprising:

[0009] Identifying an image processing operator for implementing the image processing operation represented by each first image operation node, wherein the first image operation node is: an image operation node included in the first image processing operation linked list;

[0010] Loading the identified image processing operator from pre-packaged image processing operators, instantiating the loaded image processing operator to obtain an instantiated operator;

[0011] Determining a calling order of the instantiation operators according to pointer directions between the first image operation nodes;

[0012] In accordance with the calling sequence, each instantiated operator is called in sequence to process the operator input image, and the first position of the task target in the image to be processed captured by the camera in the image coordinate system is obtained, wherein the operator input image of the first instantiated operator in the calling sequence is: the image to be processed, and the operator input images of the remaining instantiated operators except the first instantiated operator are: the image output after the image processing by the forward adjacent instantiated operator of the instantiated operator;

[0013] Based on the first position and the calibration relationship, a second position of the task target in the robot arm base coordinate system is determined, wherein the calibration relationship is: a transformation relationship between the image coordinate system and the robot arm base coordinate system.

[0014] In a second aspect, an embodiment of the present application provides a target positioning device, the device comprising:

[0015] An operator identification module, configured to identify an image processing operator for implementing the image processing operation represented by each first image operation node, wherein the first image operation node is an image operation node included in the first image processing operation linked list;

[0016] An operator instantiation module is used to load the identified image processing operator from pre-packaged image processing operators, instantiate the loaded image processing operator, and obtain an instantiated operator;

[0017] A calling sequence determining module, configured to determine the calling sequence of the instantiation operators according to the pointer directions between the first image operation nodes;

[0018] A first position obtaining module is configured to sequentially call each instantiated operator to process an operator input image in the calling order, and obtain a first position of a task target in the image to be processed captured by the camera in the image coordinate system, wherein the operator input image of the first instantiated operator in the calling order is: the image to be processed, and the operator input images of the remaining instantiated operators except the first instantiated operator are: the image output after image processing by the forward adjacent instantiated operator of the instantiated operator;

[0019] The second position determination module is used to determine the second position of the task target in the robot arm base coordinate system based on the first position and the calibration relationship, wherein the calibration relationship is: the transformation relationship between the image coordinate system and the robot arm base coordinate system.

[0020] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0021] Memory for storing computer programs;

[0022] The processor is configured to implement the method described in the first aspect when executing the program stored in the memory.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0024] In a fifth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect.

[0025] As can be seen from the above, when applying the solution provided in the embodiments of this application to perform target positioning, the image processing operators used for image processing and the execution order of the image processing operators can be determined by parsing the first image processing operation linked list. In this way, after obtaining instantiated operators through operator loading and instantiation, each instantiated operator is called in sequence to perform image processing. Based on the processing results, the position of the task target in the robot arm base coordinate system can be obtained, thereby locating the task target of the robot arm.

[0026] In addition, it can be seen that in the solution provided by the embodiment of the present application, the image processing process is described by an image processing operation chain list. In this way, when the positioning requirements of the task target change and the image processing process needs to be changed, the user selects and combines the image processing nodes according to the actual needs, and the electronic device can generate an image processing operation chain list that meets the requirements, and then execute the new image processing process based on the image processing operation chain list to locate the target. That is, the electronic device can efficiently and quickly generate an image processing operation chain list that implements the image processing process according to the actual positioning requirements, without the user having to rewrite the code of the image processing process, and has a strong adaptability to various specific positioning scenarios.

[0027] Of course, it is not necessary to achieve all the advantages described above at the same time when implementing any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0029] Figure 1 A schematic diagram of a robotic arm performing a task according to an embodiment of the present application;

[0030] Figure 2 A schematic diagram of the flow of the first target positioning method provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of a second target positioning method according to an embodiment of the present application;

[0032] Figure 4 A schematic diagram of an operator data transmission relationship provided in an embodiment of the present application;

[0033] Figure 5 A flowchart of a visual calibration method provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of a calibration point provided in an embodiment of the present application;

[0035] Figure 7 A schematic structural diagram of a first target positioning device provided in an embodiment of the present application;

[0036] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0038] First, the application scenarios of the solutions provided in the embodiments of the present application are introduced.

[0039] The application scenario of the solution provided in the embodiment of the present application is a scenario in which a robotic arm locates a task target in a working environment with the help of the visual function provided by an image acquisition device, so as to perform a set action on the task target.

[0040] Among them, the above-mentioned image acquisition device is a camera. The image acquisition device can be installed on the robotic arm or at a set position in the working environment. It is only necessary to ensure that the task target is within the field of view of the image acquisition device; the above-mentioned task target can be a workpiece, commodity, package, etc. in the production line; the above-mentioned set action can be a grasping, adsorption, assembly, measurement and other actions for the task target, which can be flexibly set according to actual task requirements. The above-mentioned set action can be specifically implemented by the end effector in the robotic arm (such as a gripper, suction cup, cutting tool, welding tool, etc.).

[0041] like Figure 1As shown, an image acquisition device 102 is installed on the robot arm 101, and the image acquisition device 102 acquires images of the workpiece 103 in the production line. The acquired images are used to locate the workpiece 103; the robot arm controls the gripper 1011 to grab the workpiece 103 according to the positioning result, and then places the workpiece 103 to the set position in the working environment.

[0042] It should be noted that Figure 1 The illustrated scenarios are merely examples for ease of understanding and do not constitute limitations on the embodiments of this application. For example, the image acquisition device 102 can be mounted via a fixed bracket in the space above or below the robotic arm. Furthermore, after the gripper 1011 grasps the workpiece 103, it can perform a predetermined assembly operation on the workpiece 103 and then return the assembled workpiece to the production line. All of these are reasonable options.

[0043] Next, the execution entity of the solution provided in the embodiment of the present application is introduced.

[0044] The execution subject of the solution provided in the embodiment of the present application is any electronic device with data processing, storage, communication and other functions, specifically a background control device or the robotic arm itself.

[0045] Next, the execution timing of the solution provided in the embodiment of the present application is introduced.

[0046] The timing for executing the solution provided in the embodiment of the present application is when the robotic arm needs to locate the task target more accurately based on the image captured by the camera.

[0047] For example, a worker can pre-set the approximate location of a task target (a photo point). The electronic device will then control the robotic arm's end effector to move to this photo point, which can also be called the location that triggers image acquisition. Once the end effector reaches the photo point, the robotic arm is considered ready to operate on the task target and needs to locate the target more precisely. At this point, the electronic device can control the camera to capture an image and begin executing this solution, locating the task target based on the image captured by the camera, allowing the end effector to accurately move to the task target's location.

[0048] The target positioning solution provided in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0049] See also Figure 2 , which is a flow chart of the first target positioning method provided in an embodiment of the present application, the above method includes the following steps S201-S205.

[0050] Step S201: Identify an image processing operator for implementing the image processing operation represented by each first image operation node.

[0051] The first image operation node is: an image operation node included in the first image processing operation linked list.

[0052] First, the first image processing operation linked list is introduced.

[0053] The first image processing operation linked list includes multiple first image operation nodes, each representing a specific image processing operation. The directions of the pointers between the first image operation nodes represent the execution order of the corresponding image processing operations. This indicates that the first image processing operation linked list overall describes the image processing operations to be performed on an image and the logical order in which these operations are executed.

[0054] The above-mentioned first image processing operation list can be generated by the electronic device in response to the user's selection and combination operations on the image operation nodes. In this way, the user can flexibly select and combine the image operation nodes according to the working environment of the robotic arm and the actual task requirements to generate the first image processing operation list; the above-mentioned first image processing operation list can also be an image processing operation list selected by the user from the image processing operation list stored in the electronic device.

[0055] Then we introduce image processing operators.

[0056] Image processing operators are operators pre-packaged and stored in electronic devices, and are used to perform various image processing operations on images. The embodiments of this application do not specifically limit this. In one case, based on the image processing operations they are responsible for, image processing operators can generally be divided into preprocessing operators and target positioning operators. Among them, the above-mentioned image preprocessing operators may include image format conversion operators, image enhancement operators, image binarization operators, image morphological transformation operators, etc.; the above-mentioned target positioning operators may include template matching operators, circle finding operators, line detection operators, etc.

[0057] Specifically, according to the set correspondence between image operation nodes and image processing operators, the image processing operator corresponding to the first image operation node can be determined as the image processing operator for implementing the image processing operation represented by the first image operation node.

[0058] Step S202: loading the identified image processing operator from pre-packaged image processing operators, instantiating the loaded image processing operator to obtain an instantiated operator.

[0059] Specifically, an operator resource can be created and registered for each first image operation node in the first image processing operation linked list, and the operator function of the image processing operator corresponding to each first image operation node can be identified; then, steps such as creating an operator tool, creating resource memory, registering the operator, and binding a callback function are executed to instantiate the above-mentioned image processing operator to obtain an instantiated operator.

[0060] Step S203: Determine the calling order of the instantiation operators according to the pointer directions between the first image operation nodes.

[0061] As explained in step S201 above, the pointer directions between the first image operation nodes represent the execution order of their corresponding image processing operations. Since image processing operations correspond one-to-one to image processing operators, it can be assumed that image operation nodes correspond one-to-one to image processing operators, and thus image operation nodes also correspond one-to-one to instantiated operators. Therefore, the execution order of the first image operation nodes can be used to determine the order in which instantiated operators are called.

[0062] For example, if the pointer direction between the first image operation node P1 and the first image operation node P2 is from P1 to P2 (ie, P1→P2), the calling order between the instantiation operators corresponding to P1 and P2 is: call P1 first, then call P2.

[0063] Step S204: calling each instantiated operator in turn to process the operator input image in the calling order, and obtaining the first position of the task target in the image to be processed captured by the camera in the image coordinate system.

[0064] Among them, the operator input image of the first instantiated operator in the calling order is: the image to be processed captured by the camera. As mentioned above, after the end effector moves to the photo-taking point, the camera can capture the image and obtain the image to be processed; the operator input image of the remaining instantiated operators except the first instantiated operator is: the image output after image processing by the forward adjacent instantiated operator of the instantiated operator.

[0065] The operator input images of all instantiated operators except the first instantiated operator can also be called intermediate images. For example, the calling order of instantiated operators OP1, OP2, and OP3 is: OP1 → OP2 → OP3. OP1 is the first instantiated operator in the calling order, and the image to be processed is the operator input image of OP1. OP1 is first called to process the image to be processed, obtaining intermediate image I1, which is also the operator input image of OP2. OP2 is called to process intermediate image I1, obtaining intermediate image I2, which is also the operator input image of OP3. Finally, OP3 is called to process intermediate image I2, obtaining the final processing result, which is the first position of the task target in the image to be processed in the image coordinate system.

[0066] Among them, the target positioning operator in the instantiation operator can identify the task target and determine the first position of the task target in the image to be processed.

[0067] Step S205: Based on the first position and the calibration relationship, determine the second position of the task target in the robot arm base coordinate system.

[0068] The calibration relationship is the transformation between the image coordinate system and the robot base coordinate system. The robot base coordinate system is a three-dimensional coordinate system based on the actual position of the robot. For example, it can be a coordinate system fixed to the base of the robot, with a point on the base as the origin. The coordinate axis directions follow certain standard conventions, such as the X axis pointing horizontally to the right, the Y axis pointing horizontally forward, and the Z axis pointing vertically upward.

[0069] The above calibration relationship can be obtained in advance, specifically by contour point calibration or calibration plate calibration. Among them, contour point calibration only requires the staff to teach the calibration center point and calibration corner point. See the following for details. Figure 5 The introduction in the illustrated embodiment will not be described in detail here; calibration plate calibration refers to visual calibration using a calibration plate pre-placed in the working scene of the robotic arm, and the specific method will not be repeated here.

[0070] Among them, by substituting the first position into the above calibration relationship, the first position can be converted into the second position in the robot arm base coordinate system.

[0071] As can be seen from the above, when applying the solution provided in the embodiments of this application to perform target positioning, the image processing operators used for image processing and the execution order of the image processing operators can be determined by parsing the first image processing operation linked list. In this way, after obtaining instantiated operators through operator loading and instantiation, each instantiated operator is called in sequence to perform image processing. Based on the processing results, the position of the task target in the robot arm base coordinate system can be obtained, thereby locating the task target of the robot arm.

[0072] In addition, it can be seen that in the solution provided by the embodiment of the present application, the image processing process is described by an image processing operation chain list. In this way, when the positioning requirements of the task target change and the image processing process needs to be changed, the user selects and combines the image processing nodes according to the actual needs, and the electronic device can generate an image processing operation chain list that meets the requirements, and then execute the new image processing process based on the image processing operation chain list to locate the target. That is, the electronic device can efficiently and quickly generate an image processing operation chain list that implements the image processing process according to the actual positioning requirements, without the user having to rewrite the code of the image processing process, and has a strong adaptability to various specific positioning scenarios.

[0073] In one embodiment of the present application, the first image processing operation linked list is recorded in the first image processing scheme. In this case, the electronic device can obtain image processing flow editing information and generate the first image processing scheme based on the image processing flow editing information.

[0074] The image processing flow editing information can be understood as operation information triggered by a user on an interactive interface. The interactive interface can be provided by an electronic device or a client deployed in a teach pendant. If the interactive interface is provided by the client, the client can send the image processing flow editing information to the electronic device.

[0075] Generating the first image processing solution includes two cases: creating a new solution and copying a solution. They are described below:

[0076] In the first case, if the image processing configuration method indicated by the image processing flow editing information is new scheme creation, the electronic device can generate a first image processing operation linked list based on the image operation nodes and the execution order information of the image operation nodes included in the image processing flow editing information, and obtain a first image processing scheme including the first image processing operation linked list.

[0077] Wherein, when the above-mentioned image operation nodes and the execution sequence information of the image operation nodes are known, pointers can be established between corresponding image operation nodes according to the execution sequence information to obtain the first image processing operation linked list.

[0078] In the second case, if the image processing configuration method indicated by the image processing flow editing information is scheme copying, the second image processing operation linked list recorded in the second image processing scheme indicated by the image processing flow editing information is copied as the first image processing operation linked list to obtain the first image processing scheme including the first image processing operation linked list.

[0079] The second image processing solution is an image processing solution stored in the electronic device.

[0080] Specifically, the interactive interface of the electronic device or the interactive interface of the client can display the image processing process represented by the image processing solution stored in the electronic device. In this way, the user can determine whether each image processing solution meets the actual task requirements based on the displayed image processing process. If it is determined that a certain image processing solution can meet the task requirements, the above image processing solution can be selected. In this case, the electronic device can copy the second image processing operation chain list recorded in the second image processing solution selected by the user to obtain the first image processing solution including the first image processing operation chain list.

[0081] In this embodiment, the electronic device can generate an image processing solution by reconstructing the image processing solution or selecting an existing image processing solution according to different user operations. For the user, when dealing with positioning tasks in different scenarios, the user can flexibly choose the method of configuring the image processing solution according to the actual situation. Specifically, when configuring the image processing solution, you can either create a new solution to meet the actual task requirements or copy an existing solution, which reduces the difficulty of configuring the image processing solution and improves the configuration efficiency. It can be seen that through multiple solution configuration methods, the flexibility and configuration efficiency when creating the image processing solution are improved.

[0082] In one embodiment of the present application, Figure 2 Based on the illustrated embodiment, the electronic device is also pre-configured with a region of interest (ROI) and / or ROI inheritance information of the first image operation node. In this case, when each instantiated operator is called sequentially in the calling order to process the operator input image in step S204, the target ROI of each instantiated operator can be first determined, and then each instantiated operator can be called sequentially to process the target ROI corresponding to each instantiated operator in the operator input image. In view of the above situation, the embodiment of the present application provides a second target positioning method.

[0083] See also Figure 3 , which is a flow chart of the second target positioning method provided in an embodiment of the present application, the above method includes the following steps S301-S307.

[0084] Step S301: Identify an image processing operator for implementing the image processing operation represented by each first image operation node.

[0085] Step S302: loading the identified image processing operator from pre-packaged image processing operators, instantiating the loaded image processing operator to obtain an instantiated operator.

[0086] Step S303: Determine the calling order of the instantiation operators according to the pointer directions between the first image operation nodes.

[0087] The above steps S301 to S303 are the same as the above Figure 2 In the illustrated embodiment, steps S201 to S203 are the same and will not be described in detail here.

[0088] Step S304: If the ROI inheritance information configured in the image operation node corresponding to the instantiation operator represents an inherited ROI, the first ROI is used as the target ROI of the instantiation operator.

[0089] The first ROI is: the ROI output by the image operation node corresponding to the forward adjacent instantiation operator of the instantiation operator.

[0090] That is, for an instantiation operator that has inherited ROI, the ROI output by its forward adjacent instantiation operator can be used as its own target ROI, which is equivalent to the instantiation operator "inheriting" the ROI output by the forward adjacent instantiation operator.

[0091] Step S305: If the image operation node corresponding to the instantiation operator is not configured with ROI inheritance information or the configured ROI inheritance information indicates that the ROI is not inherited, the second ROI is used as the target ROI of the instantiation operator.

[0092] The second ROI is: the ROI configured by the image operation node corresponding to the instantiation operator.

[0093] That is, for an instantiated operator that is set not to inherit ROI, its own configured ROI can be used as the target ROI, which is equivalent to the instantiated operator applying its own configured ROI without "inheriting" the ROI output by the adjacent instantiated operator.

[0094] In one embodiment of the present application, the electronic device is further provided with ROI correction indication information for each image operation node. In this case, for an instantiation operator corresponding to an image operation node that is set not to inherit the ROI, it can also be determined whether the ROI correction indication information corresponding to the instantiation operator represents a correction of the ROI. If it represents a correction of the ROI, the second ROI is corrected based on the aforementioned first ROI, that is, the self-configured ROI is corrected based on the ROI output by the forward adjacent operator, and the corrected ROI is used as the target ROI of the instantiation operator; if it represents a non-correction of the ROI, the second ROI is used as the target ROI of the instantiation operator, that is, the self-configured ROI is used as the target ROI of the instantiation operator.

[0095] The second ROI is modified based on the first ROI, that is, the first ROI obtained according to the forward adjacent instantiation operator is adaptively modified for the second ROI configured by itself to improve the accuracy of the ROI.

[0096] For example, the instantiation operator OP1 is a positioning operator, and OP1's forward instantiation operator OP2 is also a positioning operator. The ROI1 of OP1 is set by the staff, and its accuracy is limited due to the dynamic motion of the robotic arm. After OP2 outputs ROI2, the position of ROI2 reflects the latest position of the target of interest in the image. Therefore, the ROI1 set by the staff for OP1 can be corrected based on the ROI2 output by OP2, resulting in a more accurate ROI1. It can be seen that adjusting the ROI configured by the instantiation operator itself based on the ROI output by the forward adjacent instantiation operator can improve the accuracy and rationality of the target ROI determined for the instantiation operator.

[0097] Specifically, the second ROI may be adjusted based on the first ROI in the following manner.

[0098] In one implementation, the adjustment direction and adjustment value of the second ROI may be determined based on the relative position relationship and distance between the first ROI and the second ROI, and the second ROI may be adjusted according to the determined adjustment method and adjustment value.

[0099] For example, if the first ROI is located above and right of the second ROI, and the distance between the first ROI and the second ROI is X, the adjustment mode of the second ROI can be determined to be above and right, and the adjustment value is aX, where a is a setting coefficient greater than 0 and less than or equal to 1.

[0100] In another embodiment, when the forward adjacent instantiation operator outputs the first ROI, it also outputs the first feature point of the target of interest in the first ROI. In this case, the position of the second ROI can be adjusted according to the position of the first feature point in combination with the set correction rule.

[0101] For example, if the first feature points are mostly distributed above the second ROI, the position of the second ROI is moved upward according to the correction rule.

[0102] It should be noted that if the image operation node is not configured with ROI and ROI inheritance information, the entire image area can be used as its target ROI.

[0103] Step S306: calling each instantiation operator in sequence according to the calling order, processing the target ROI corresponding to each instantiation operator in the operator input image, and obtaining the first position of the task target in the image to be processed captured by the camera in the image coordinate system.

[0104] Step S307: Based on the first position and the calibration relationship, determine the second position of the task target in the robot arm base coordinate system.

[0105] As can be seen from the above, in this embodiment, when calling each instantiated operator to process the operator input image, the target ROI of the instantiated operator can be determined first, and then the instantiated operator can be called to process the ROI in the operator input image in a targeted manner, thereby reducing the computing resources consumed by image processing; furthermore, by setting the ROI and ROI inheritance status information for each image operation node, the ROI required to be processed by each instantiated operator can be flexibly set, thereby improving the flexibility of image processing.

[0106] In one embodiment of the present application, the instantiated operators include: a first type of operators and a second type of operators, wherein the first type of operators are: instantiated operators used to implement image preprocessing operations, and the second type of operators are: instantiated operators used to implement target positioning operations.

[0107] That is, in this case, the first image operation node includes: a node for implementing image preprocessing and a node for implementing target positioning. Thus, the instantiated operator obtained based on the first image operation node includes a first type of operator for implementing image preprocessing operations (hereinafter referred to as preprocessing operators) and a second type of operator for implementing target positioning operations (hereinafter referred to as positioning operators).

[0108] The output of the preprocessing operator includes: an image obtained after the preprocessing operator performs image processing and a first ROI output by a forward adjacent instantiation operator of the preprocessing operator.

[0109] Preprocessing operators generally perform binarization, image enhancement, and other processing on the entire input image of the operator responsible for processing, and then output the image with the changed content. However, preprocessing operators generally do not perform target detection operations and therefore generally do not output new ROIs. Therefore, in order to enable subsequent instantiation operators to know the ROI in the image, in addition to outputting the changed image, the preprocessing operator can also output the ROI output by its forward adjacent instantiation operator. In other words, the ROI output by its forward adjacent instantiation operator is passed on, which can be called a transparent ROI.

[0110] The output of the positioning operator includes: the image output by the forward adjacent instantiation operator and the ROI obtained after the positioning operator performs image processing.

[0111] A localization operator typically performs target detection on the input image of the operator it is processing and then outputs a new ROI. However, the overall image content is generally not altered during the execution of the localization operator, and thus a new image with altered content is generally not output. Therefore, to enable subsequent instantiation operators to obtain image data, the localization operator can output not only the new ROI but also the image output by its preceding adjacent instantiation operator. This is called a pass-through image, as it passes the output image of the preceding adjacent instantiation operator forward.

[0112] The following combination Figure 4 , introduced through a specific example.

[0113] See also Figure 4 The input of the positioning operator O1 includes ROI1 and image P1. After the positioning operator O1 locates the target on image P1, it obtains a new ROI2. Then, the positioning operator O1 outputs ROI2 to the preprocessing operator O2 and transparently transmits the image P1. The preprocessing operator O2 preprocesses the image P1 to obtain a new image P2. Then, the preprocessing operator O2 outputs the new image P2 to the positioning operator O3 and transparently transmits ROI2.

[0114] As can be seen from the above, in this embodiment, the first type of operator used to implement image preprocessing operations not only outputs an image but also transparently transmits the ROI output by the preceding adjacent instantiated operator; the second type of operator used to implement target positioning operations not only outputs an ROI but also transparently transmits the image output by the preceding adjacent instantiated operator. In this way, both the first and second type of operators output an image and an ROI, allowing subsequent instantiated operators to always obtain image data and ROI information. This prevents data transmission or information loss when the instantiated operators perform image processing sequentially, improves the rationality of the solution, and ensures the accuracy of image processing operations.

[0115] The following combination Figure 5 , the contour calibration method mentioned above is introduced.

[0116] See also Figure 5 , is a flow chart of a calibration relationship obtaining method provided in an embodiment of the present application, the method includes the following steps S501-S503.

[0117] Step S501: Generate a calibration path based on the set positions of the calibration points.

[0118] The position of the above-mentioned calibration point is the spatial position of the index fixed point in the robot arm base coordinate system, which can be obtained through teaching by the staff.

[0119] Specifically, the calibration path may be generated in the following manner.

[0120] In one case, the locations of the calibration points set include: a first location of a calibration center point and a second location of a calibration corner point. The calibration center point is a calibration point located at the center of the calibration area covered by the calibration path to be generated, and the calibration corner points are calibration points located at the corners of the calibration area. If the locations of the calibration center point and calibration corner points are known, the locations of the remaining calibration points in the calibration path can be calculated according to the following steps A-B to obtain a complete calibration path:

[0121] Step A: Based on the first position, the second position and the number of calibration points recorded in the set calibration point distribution information, determine the third position of the remaining translation calibration points (for ease of expression, referred to as the first calibration point below) except the calibration center point and the calibration corner point and the fourth position of the rotation calibration point.

[0122] The number of calibration points records the total number of translation calibration points, such as 9, and the translation calibration points include: calibration center point, calibration corner point and first calibration point; the number of calibration points can also record the number of rotation calibration points, such as 1.

[0123] Specifically, based on the distance between the first position of the calibration center point and the second position of the calibration corner point, and combined with the calibration point distribution shape in the calibration point distribution information, the positions of the first calibration point and the rotation calibration point can be determined. Figure 6 Taking the calibration point distribution shape as a square as an example, the methods of determining the position of the first calibration point and the rotation calibration point are introduced respectively by giving examples, wherein, Figure 6 The circles in the figure represent the calibration points, and calibration point 1 and calibration point 2 are the calibration center point and calibration corner point respectively.

[0124] For the first calibration point:

[0125] First, calculate the direction vector S1 from calibration point 1 to calibration point 2. Given that calibration point 1 is the calibration center point and the calibration point distribution is a square, S1 is half the diagonal of the square. Rotate S1 90° to obtain vector S2. Calculate the difference between vector S2 and the calibration center point to obtain the position of the second calibration corner point (calibration point 4). Similarly, rotate S1 180° and 270° to obtain the positions of the third and fourth calibration corner points (calibration point 6 and calibration point 8). Next, calculate the midpoint coordinates between each pair of calibration corner points to obtain the positions of the four calibration points (calibration point 3, calibration point 5, calibration point 7, and calibration point 9). At this point, the positions of all nine translation calibration points are determined. Figure 4 shows the positions of each translation calibration point, indicating that calibration points 1 to 9 are translation calibration points.

[0126] For rotated calibration points:

[0127] The calibration point distribution information may record the position of the rotation calibration point in the calibration point distribution shape. For example, if the position of the rotation calibration point in the calibration point distribution shape recorded in the calibration point distribution information is the calibration center point, the position of the calibration center point may be directly used as the position of the rotation calibration point, i.e. Figure 6 The calibration point 1 in is the rotation calibration point. It can be seen that calibration point 1 is both a translation calibration point and a rotation calibration point.

[0128] Step B: Generate a calibration path including translation calibration points and rotation calibration points according to the traversal order information recorded in the calibration point distribution information, and generate control information for controlling the rotation angle of the end effector at the rotation calibration point based on the set rotation angle.

[0129] The calibration point distribution information records the distribution shape of the calibration points and can also record the traversal order of each calibration point in the calibration point distribution shape. Therefore, the direction information between each translation calibration point and rotation calibration point can be generated according to the traversal order to obtain the calibration path.

[0130] Continuing the example above where the calibration points are distributed in a square shape, the generated path is as follows Figure 6As shown, Figure 6 In the figure, the arrows starting from the translation calibration point indicate the traversal order between calibration points, and the arrows starting from the rotation calibration point indicate the corresponding rotation angle of the rotation calibration point. For calibration point 1, it is both a translation calibration point and a rotation calibration point. Among the three arrows starting from calibration point 1, the arrow pointing to the upper right indicates the traversal order, and the two arrows pointing to the upper left and lower left indicate the two rotation angles corresponding to calibration point 1.

[0131] The control information is used to control the rotation angle of the end effector of the robot arm at the rotation calibration point. After the control information is generated, it can be added to the calibration path as the attribute information of the rotation point, or it can be stored separately.

[0132] In this embodiment, the electronic device can automatically generate a calibration path based on the set calibration center point and calibration corner points, and control the end effector movement and camera image acquisition based on the calibration path to complete the visual calibration. As can be seen, for the staff, they only need to teach the calibration center point and calibration corner points, and the electronic device can automatically generate the calibration path and complete the calibration steps. The calibration process does not require a calibration plate, which simplifies the visual calibration process and improves visual calibration efficiency.

[0133] In another case, the electronic device may be pre-configured with the positions and traversal order information of all calibration points required for calibration, so that a calibration path can be generated based on the above complete information.

[0134] Step S502: controlling the movement of the end effector of the robotic arm according to the calibration path, and in response to the end effector moving to the position of the calibration point included in the calibration path, controlling the camera to perform image acquisition to obtain a calibration image.

[0135] In this step, whenever the electronic device determines that the end effector moves to the position of the calibration point included in the calibration path, the camera can be controlled to capture an image to obtain a calibration image.

[0136] Specifically, when the calibration points include translation calibration points and rotation calibration points, the electronic device controls the camera to capture images in the following situations:

[0137] If it is determined that the end effector moves to the position of the translation calibration point included in the calibration path, the camera is directly controlled to capture the image; if it is determined that the end effector moves to the position of the rotation calibration point included in the calibration path, the end effector is controlled to rotate to a first angle, and then the camera is controlled to capture the image, and the above-mentioned first angle is the rotation angle indicated by the above-mentioned control information.

[0138] In this embodiment, the calibration path includes two types of calibration points: translational and rotational. For translational calibration points, the electronic device controls the camera to capture images after determining the position of the end effector at each translational calibration point. For rotational calibration points, the electronic device first controls the end effector to rotate based on control information before controlling the camera to capture images. As can be seen, the end effector's trajectory along the calibration path includes both translational and rotational trajectories. This allows the camera to capture images containing diverse content, improving the accuracy of subsequent visual calibration based on image content.

[0139] Step S503: After determining the calibration points included in the calibration path traversed by the end effector, the robot arm and the camera are visually calibrated based on the spatial position and image position of the feature points of the set markers in the collected calibration images to obtain a calibration relationship.

[0140] The following is an introduction to the above markers:

[0141] The above-mentioned marker can be a set object with obvious features and easy-to-extract feature points, and the marker moves as the end effector moves.

[0142] In one case, the marker can be an object fixed to or grasped by the end effector, such as a rectangular plate or a reflective ball. In this case, feature points of the object can be extracted, such as the center or corner of the rectangular plate as the feature point of the rectangular plate, or the center of the reflective ball as the feature point of the reflective ball.

[0143] Alternatively, if the end effector itself has distinct features, the marker can be the end effector itself. In this case, feature points of the end effector can be extracted. For example, if the end effector is a spray gun, feature points of a specific location on the gun (such as the nozzle) can be extracted.

[0144] In this embodiment, the camera may be fixedly mounted in a working environment, and the marker is within the field of view of the camera. Therefore, the calibration image captured by the camera includes the marker.

[0145] The following describes a method for determining the image position and spatial position of the feature points of the marker in each calibration image.

[0146] Since the marker moves with the movement of the end effector, as the end effector moves along the calibration path to the position of each calibration point, multiple calibration images with different marker positions can be collected. Using a set feature point extraction algorithm, feature points of markers with different positions can be extracted from each calibration image, and the image position of the extracted feature points can be determined. Before feature point extraction, the calibration image can be subjected to image preprocessing to improve extraction accuracy. In addition, since the position of each calibration point in the robot arm base coordinate system is known, the spatial position of the feature point of the marker in each calibration image in the robot arm base coordinate system can be obtained based on the relative position relationship between the pre-measured feature point and each calibration point.

[0147] After obtaining the image position and spatial position of the feature points in each calibration image, visual calibration can be performed. Visual calibration is performed to obtain the conversion relationship between the robot base coordinate system and the image coordinate system of the image captured by the camera.

[0148] Specifically, the intrinsic parameter matrix of the camera can be obtained first. The above intrinsic parameter matrix is ​​used to describe the geometric characteristics inside the camera, is related to the hardware parameters, can be obtained according to the factory parameters of the camera, and is used to project the points in the camera coordinate system to the image plane. Based on the intrinsic parameter matrix, the first transformation relationship between the image coordinate system and the camera coordinate system can be obtained; then, based on the spatial position and image position of each feature point, the extrinsic parameter matrix of the camera is solved using algorithms such as the Perspective-n-Point (PnP) algorithm. The above extrinsic parameter matrix is ​​also the second transformation relationship between the camera coordinate system and the robotic arm base space coordinate system; after obtaining the above first transformation relationship and the second transformation relationship, the transformation relationship between the robotic arm base coordinate system and the image coordinate system can be obtained through a simple transformation of the image coordinate system → camera coordinate system → robotic arm base coordinate system.

[0149] In one embodiment of the present application, when the calibration path includes both translation calibration points and rotation calibration points, the conversion relationship between the robot arm base coordinate system and the image coordinate system can be calculated based on the spatial position and image position of the feature points in the calibration image captured by the camera at each translation calibration point using the above method; then, the conversion relationship between the robot arm base coordinate system and the image coordinate system can be corrected using the spatial position and image position of the feature points in the calibration image captured by the camera after the end effector rotates at the rotation calibration point.

[0150] As can be seen from the above, in this embodiment, the electronic device can generate a calibration path based on the set calibration point position, and then control the movement of the end effector of the robotic arm according to the calibration path. When the end effector moves to the position of the calibration point included in the calibration path, the camera is controlled to perform image acquisition to obtain each calibration image. The electronic device can autonomously identify the feature points of the marker in the calibration image, and finally, based on the spatial position and image position of the feature point of the set marker in each calibration image collected, the robotic arm and the camera can be visually calibrated to obtain a calibration relationship. It can be seen that in the above calibration process, the user only needs to teach the calibration point position in advance, and the electronic device can autonomously implement the subsequent calibration process, which simplifies the calibration process and allows the user to easily complete the calibration work between the camera and the robotic arm.

[0151] In one embodiment of the present application, the above-mentioned calibration process can be stored in the electronic device in the form of a calibration scheme, so that the electronic device can respond to external calibration instructions, load and execute the calibration scheme, and control the robotic arm and camera to perform visual calibration.

[0152] In one embodiment of the present application, the execution process in the calibration solution and the execution process in the aforementioned image processing solution can be copied to each other.

[0153] For example, as we've seen in the previous section, when electronic devices control a robotic arm and camera for visual calibration, they can perform image processing steps like image preprocessing and feature extraction to identify feature points. This means that the calibration solution needs to include a record of this image processing process. Similarly, the image processing solution for target positioning also includes a record of this process. Therefore, both the calibration solution and the image processing solution need to include a record of this process.

[0154] In this case, the image processing flow in the image processing solution can be copied and used in the calibration solution; or the image processing flow in the calibration solution can be copied and used in the image processing solution.

[0155] In one embodiment of the present application, after the electronic device completes the above-mentioned calibration process and obtains the calibration relationship, it can send the calibration relationship and the translation error and rotation error generated during the calibration process to the client deployed in the aforementioned teaching pendant. The calibration relationship can be sent in the form of a homogeneous transformation matrix; the above-mentioned translation error and rotation error can be obtained by substituting the image position of each feature point into the calibration relationship and comparing the obtained result with the spatial position of each feature point.

[0156] In this way, the user can browse the calibration relationship and the above-mentioned calibration error obtained from this calibration on the client, and evaluate the calibration results. If the calibration results are considered to be highly accurate, they can be applied directly; otherwise, the electronic device can be made to re-execute the calibration process by changing the calibration path, changing the marker, changing the image processing algorithm, etc., to obtain more accurate calibration results.

[0157] In one embodiment of the present application, after obtaining the calibration results, if the accuracy of the calibration results is low, the electronic device can obtain recognition result evaluation information of each calibration image; then, based on the recognition result evaluation information, it can adjust the calibration path and / or adjust the visual processing parameters used when extracting feature points.

[0158] The above-mentioned recognition result evaluation information may include the completeness of the markers in the calibration image, the accuracy of the feature points extracted from the calibration image, etc., which are respectively introduced below with examples.

[0159] For example, if it is determined that the markers in the images corresponding to certain calibration points are incomplete, this indicates that the end effector moved the marker beyond the field of view of the image acquisition device when moving to certain calibration points. In this case, the calibration path can be changed so that the end effector moves the marker to each calibration point included in the new calibration path within the field of view of the image acquisition device. This ensures that the markers in the images corresponding to each calibration point are complete, which can improve the accuracy of subsequent feature point extraction and, therefore, the accuracy of calibration.

[0160] For example, if it is determined that the position error of the feature points in the image recognition results corresponding to certain calibration points is large, that is, a large error occurs in the stage of extracting the feature points of the marker, in this case, the maximum number of matches required for extracting image feature points, the number of calipers (ROI used to scan edge points), edge polarity and other parameters can be adjusted so that the feature points can be identified more accurately based on the adjusted parameters.

[0161] In one embodiment of the present application, the electronic device can display the images and image recognition results collected when the robotic arm moves to each calibration point in the interactive interface, so that the user can control the electronic device to adjust the calibration path and / or change the visual processing parameters according to the displayed information.

[0162] In the technical solution of this application, the operations involved in obtaining, storing, using, processing, transmitting, providing and disclosing user personal information are all carried out with the user's authorization.

[0163] Corresponding to the above-mentioned target positioning method, an embodiment of the present application also provides a target positioning device.

[0164] See also Figure 7, is a schematic diagram of the structure of a target positioning device provided in an embodiment of the present application, the device includes the following modules:

[0165] An operator identification module 701 is configured to identify an image processing operator for implementing an image processing operation represented by each first image operation node, wherein the first image operation node is an image operation node included in a first image processing operation linked list;

[0166] An operator instantiation module 702 is configured to load an identified image processing operator from pre-packaged image processing operators, instantiate the loaded image processing operator, and obtain an instantiated operator;

[0167] A calling order determining module 703 is configured to determine the calling order of the instantiation operators according to the pointer directions between the first image operation nodes;

[0168] A first position obtaining module 704 is configured to sequentially call each instantiated operator to process the operator input image in the calling order, and obtain the first position of the task target in the image to be processed captured by the camera in the image coordinate system, wherein the operator input image of the first instantiated operator in the calling order is: the image to be processed, and the operator input images of the remaining instantiated operators except the first instantiated operator are: the image output after image processing by the instantiated operator's forward adjacent instantiated operator;

[0169] The second position determination module 705 is used to determine the second position of the task target in the robot arm base coordinate system based on the first position and the calibration relationship, wherein the calibration relationship is: the transformation relationship between the image coordinate system and the robot arm base coordinate system.

[0170] As can be seen from the above, when applying the solution provided in the embodiments of this application to perform target positioning, the image processing operators used for image processing and the execution order of the image processing operators can be determined by parsing the first image processing operation linked list. In this way, after obtaining instantiated operators through operator loading and instantiation, each instantiated operator is called in sequence to perform image processing. Based on the processing results, the position of the task target in the robot arm base coordinate system can be obtained, thereby locating the task target of the robot arm.

[0171] In addition, it can be seen that in the solution provided by the embodiment of the present application, the image processing process is described by an image processing operation chain list. In this way, when the positioning requirements of the task target change and the image processing process needs to be changed, the user selects and combines the image processing nodes according to the actual needs, and the electronic device can generate an image processing operation chain list that meets the requirements, and then execute the new image processing process based on the image processing operation chain list to locate the target. That is, the electronic device can efficiently and quickly generate an image processing operation chain list that implements the image processing process according to the actual positioning requirements, without the user having to rewrite the code of the image processing process, and has a strong adaptability to various specific positioning scenarios.

[0172] In one embodiment of the present application, the first image operation node is pre-configured with: a region of interest ROI and / or ROI inheritance information; the first position obtaining module 704 includes:

[0173] A first ROI determination module is configured to use the first ROI as a target ROI of the instantiation operator if the ROI inheritance information configured by the first image operation node corresponding to the instantiation operator represents an inherited ROI, wherein the first ROI is: an ROI output by the first image operation node corresponding to the forward adjacent instantiation operator of the instantiation operator;

[0174] a second ROI determination module, configured to use a second ROI as a target ROI of the instantiation operator if the first image operation node corresponding to the instantiation operator is not configured with ROI inheritance information or the configured ROI inheritance information indicates that the ROI is not inherited; wherein the second ROI is: the ROI configured by the first image operation node corresponding to the instantiation operator;

[0175] The position acquisition submodule is used to call each instantiation operator in sequence according to the calling order, process the target ROI corresponding to each instantiation operator in the operator input image, and obtain the first position of the task target in the image to be processed captured by the camera in the image coordinate system.

[0176] As can be seen from the above, in this embodiment, when calling each instantiated operator to process the operator input image, the target ROI of the instantiated operator can be determined first, and then the instantiated operator can be called to process the ROI in the operator input image in a targeted manner, thereby reducing the computing resources consumed by image processing; furthermore, by setting the ROI and ROI inheritance status information for each image operation node, the ROI required to be processed by each instantiated operator can be flexibly set, thereby improving the flexibility of image processing.

[0177] In one embodiment of the present application, the following is pre-configured: ROI correction instruction information of the first image operation node,

[0178] The second ROI determination module is specifically configured to, if the first image operation node corresponding to the instantiation operator is not configured with ROI inheritance information or the configured ROI inheritance information indicates that the ROI is not inherited, determine whether the ROI correction indication information configured by the first image operation node corresponding to the instantiation operator indicates a corrected ROI; if not, use the second ROI as the target ROI of the instantiation operator, wherein the second ROI is: the ROI configured by the first image operation node corresponding to the instantiation operator; if yes, correct the second ROI based on the first ROI, and use the corrected ROI as the target ROI of the instantiation operator.

[0179] It can be seen that adjusting the ROI configured by the instantiation operator itself based on the ROI output by the forward adjacent instantiation operator can improve the accuracy and rationality of the target ROI determined for the instantiation operator.

[0180] In one embodiment of the present application, the instantiated operator includes: a first type of operator and a second type of operator, wherein the first type of operator is: an instantiated operator for implementing image preprocessing operations, and the second type of operator is: an instantiated operator for implementing target positioning operations; the output of the first type of operator includes: the image obtained after the first type of operator performs image processing and the first ROI; the output of the second type of operator includes: the image output by the forward adjacent instantiated operator and the ROI obtained after the second type of operator performs image processing.

[0181] As can be seen from the above, in this embodiment, the first type of operator used to implement image preprocessing operations not only outputs an image but also transparently transmits the ROI output by the preceding adjacent instantiated operator; the second type of operator used to implement target positioning operations not only outputs an ROI but also transparently transmits the image output by the preceding adjacent instantiated operator. In this way, both the first and second type of operators output an image and an ROI, allowing subsequent instantiated operators to always obtain image data and ROI information. This prevents data transmission or information loss when the instantiated operators perform image processing sequentially, improves the rationality of the solution, and ensures the accuracy of image processing operations.

[0182] In one embodiment of the present application, the calibration relationship is obtained according to the following modules:

[0183] A calibration path generation module, used to generate a calibration path based on the positions of the set calibration points;

[0184] a calibration image acquisition module, configured to control the movement of the end effector of the robotic arm according to the calibration path, and in response to the end effector moving to the position of a calibration point included in the calibration path, control the camera to perform image acquisition to obtain a calibration image;

[0185] A calibration relationship acquisition module is used to determine that after the end effector traverses each calibration point included in the calibration path, the robot arm and the camera are visually calibrated based on the spatial position and image position of the feature point of the set marker in each captured calibration image to obtain a calibration relationship, wherein the spatial position is determined based on the position of the calibration point included in the calibration path.

[0186] As can be seen from the above, in this embodiment, the electronic device can generate a calibration path based on the set calibration point position, and then control the movement of the end effector of the robotic arm according to the calibration path. When the end effector moves to the position of the calibration point included in the calibration path, the camera is controlled to perform image acquisition to obtain each calibration image. The electronic device can autonomously identify the feature points of the marker in the calibration image, and finally, based on the spatial position and image position of the feature point of the set marker in each calibration image collected, the robotic arm and the camera can be visually calibrated to obtain a calibration relationship. It can be seen that in the above calibration process, the user only needs to teach the calibration point position in advance, and the electronic device can autonomously implement the subsequent calibration process, which simplifies the calibration process and allows the user to easily complete the calibration work between the camera and the robotic arm.

[0187] In one embodiment of the present application, the positions of the calibration points include: a first position of a calibration center point and a second position of a calibration corner point, wherein the calibration center point is a calibration point located at the center of a calibration area covered by a calibration path to be generated, and the calibration corner point is a calibration point located at a corner point of the calibration area.

[0188] The calibration path generation module is specifically used to determine the third position of the first calibration point and the fourth position of the rotation calibration point other than the calibration center point and the calibration corner point for visual calibration based on the first position, the second position and the number of calibration points recorded in the set calibration point distribution information; generate a calibration path including the translation calibration point and the rotation calibration point according to the traversal order information recorded in the calibration point distribution information, and generate control information for controlling the rotation angle of the end effector at the rotation calibration point based on the set rotation angle, wherein the translation calibration point includes: the calibration center point, the calibration corner point and the first calibration point.

[0189] In this embodiment, the electronic device can automatically generate a calibration path based on the set calibration center point and calibration corner points, and control the end effector movement and camera image acquisition based on the calibration path to complete the visual calibration. As can be seen, for the staff, they only need to teach the calibration center point and calibration corner points, and the electronic device can automatically generate the calibration path and complete the calibration steps. The calibration process does not require a calibration plate, which simplifies the visual calibration process and improves visual calibration efficiency.

[0190] In one embodiment of the present application, the calibration image acquisition module is specifically used to control the camera to perform image acquisition if it is determined that the end effector moves to the position of the translation calibration point included in the calibration path; if it is determined that the end effector moves to the position of the rotation calibration point included in the calibration path, control the end effector to rotate a first angle and control the camera to perform image acquisition, wherein the first angle is: the rotation angle indicated by the control information of the rotation calibration point.

[0191] In this embodiment, the calibration path includes two types of calibration points: translational and rotational. For translational calibration points, the electronic device controls the camera to capture images after determining the position of the end effector at each translational calibration point. For rotational calibration points, the electronic device first controls the end effector to rotate based on control information before controlling the camera to capture images. As can be seen, the end effector's trajectory along the calibration path includes both translational and rotational trajectories. This allows the camera to capture images containing diverse content, improving the accuracy of subsequent visual calibration based on image content.

[0192] In one embodiment of the present application, the first image processing operation linked list is recorded in a first image processing solution, and the first image processing solution is generated according to the following modules:

[0193] An editing information obtaining module is used to obtain image processing flow editing information;

[0194] a first solution obtaining module configured to, if the image processing configuration mode indicated by the image processing flow editing information is solution creation, generate a first image processing operation linked list based on the first image operation node and execution order information of the first image operation node included in the image processing flow editing information, and obtain a first image processing solution including the first image processing operation linked list;

[0195] A second scheme obtaining module is used to copy the second image processing operation linked list recorded in the second image processing scheme indicated by the image processing process editing information as the first image processing operation linked list if the image processing configuration mode indicated by the image processing process editing information is scheme copying, and obtain a first image processing scheme including the first image processing operation linked list.

[0196] In this embodiment, the electronic device can generate an image processing solution by reconstructing the image processing solution or selecting an existing image processing solution according to different user operations. For the user, when dealing with positioning tasks in different scenarios, the user can flexibly choose the method of configuring the image processing solution according to the actual situation. Specifically, when configuring the image processing solution, you can either create a new solution to meet the actual task requirements or copy an existing solution, which reduces the difficulty of configuring the image processing solution and improves the configuration efficiency. It can be seen that through multiple solution configuration methods, the flexibility and configuration efficiency when creating the image processing solution are improved.

[0197] Corresponding to the above-mentioned target positioning method, an embodiment of the present application further provides an electronic device, a computer-readable storage medium, and a computer program.

[0198] The present application embodiment provides an electronic device, such as Figure 8 Shown, including:

[0199] Memory 801, used for storing computer programs;

[0200] The processor 802 is configured to implement the aforementioned target positioning method when executing the program stored in the memory 801 .

[0201] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 802, the communication interface, and the memory 801 communicate with each other via the communication bus.

[0202] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0203] The communication interface is used for communication between the above electronic device and other devices.

[0204] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk memory. Alternatively, the memory may be at least one storage device located away from the processor.

[0205] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0206] In another embodiment provided by the present application, a computer-readable storage medium is further provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned target positioning method is implemented.

[0207] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the above-mentioned target positioning method.

[0208] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital staff line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state drive (SSD).

[0209] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0210] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. In particular, the device, electronic device, and storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For related portions, reference can be made to the descriptions of the method embodiments.

[0211] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A target positioning method, characterized in that: The method comprises: Identifying an image processing operator for implementing the image processing operation represented by each first image operation node, wherein the first image operation node is: an image operation node included in the first image processing operation linked list; Loading the identified image processing operator from pre-packaged image processing operators, instantiating the loaded image processing operator to obtain an instantiated operator; Determining a calling order of the instantiation operators according to pointer directions between the first image operation nodes; In accordance with the calling sequence, each instantiated operator is called in sequence to process the operator input image, and the first position of the task target in the image to be processed captured by the camera in the image coordinate system is obtained, wherein the operator input image of the first instantiated operator in the calling sequence is: the image to be processed, and the operator input images of the remaining instantiated operators except the first instantiated operator are: the image output after the image processing by the forward adjacent instantiated operator of the instantiated operator; Based on the first position and the calibration relationship, a second position of the task target in the robot arm base coordinate system is determined, wherein the calibration relationship is: a transformation relationship between the image coordinate system and the robot arm base coordinate system.

2. The method according to claim 1, characterized in that Pre-configured with: the region of interest ROI and / or ROI inheritance information of the first image operation node, and calling each instantiated operator in sequence to process the operator input image according to the calling order, including: If the ROI inheritance information configured by the first image operation node corresponding to the instantiation operator represents an inherited ROI, the first ROI is used as the target ROI of the instantiation operator, wherein the first ROI is: the ROI output by the first image operation node corresponding to the forward adjacent instantiation operator of the instantiation operator; If the first image operation node corresponding to the instantiation operator is not configured with ROI inheritance information or the configured ROI inheritance information indicates that it does not inherit the ROI, the second ROI is used as the target ROI of the instantiation operator, wherein the second ROI is: the ROI configured by the first image operation node corresponding to the instantiation operator; In accordance with the calling order, each instantiation operator is called in turn to process the target ROI corresponding to each instantiation operator in the operator input image.

3. The method according to claim 2, characterized in that Pre-configured with: ROI correction indication information of the first image operation node, if the first image operation node corresponding to the instantiation operator is not configured with ROI inheritance information or the configured ROI inheritance information indicates that the ROI is not inherited, then the second ROI is used as the target ROI of the instantiation operator, including: If the first image operation node corresponding to the instantiation operator is not configured with ROI inheritance information or the configured ROI inheritance information indicates that the ROI is not inherited, determining whether the ROI correction indication information configured in the first image operation node corresponding to the instantiation operator indicates that the ROI is corrected; If not, use the second ROI as the target ROI of the instantiation operator, wherein the second ROI is: the ROI configured by the first image operation node corresponding to the instantiation operator; If yes, the second ROI is modified based on the first ROI, and the modified ROI is used as the target ROI of the instantiation operator.

4. The method according to claim 2, characterized in that The instantiation operators include: a first type of operators and a second type of operators, wherein the first type of operators are: instantiation operators for implementing image preprocessing operations, and the second type of operators are: instantiation operators for implementing target positioning operations; The output of the first type of operator includes: an image obtained after the first type of operator performs image processing and the first ROI; The output of the second type of operator includes: an image output by the forward adjacent instantiation operator and a ROI obtained after the second type of operator performs image processing.

5. The method according to claim 1, wherein The calibration relationship is obtained as follows: Generate a calibration path based on the positions of the set calibration points; controlling the end effector of the robotic arm to move according to the calibration path, and in response to the end effector moving to a position of a calibration point included in the calibration path, controlling the camera to capture an image to obtain a calibration image; After determining that the end effector traverses the calibration points included in the calibration path, the robotic arm and the camera are visually calibrated based on the spatial positions and image positions of the feature points of the set markers in the collected calibration images to obtain a calibration relationship, wherein the spatial positions are determined based on the positions of the calibration points included in the calibration path.

6. The method according to claim 5, characterized in that The positions of the calibration points include: a first position of a calibration center point and a second position of a calibration corner point, wherein the calibration center point is a calibration point located at the center of a calibration area covered by a calibration path to be generated, and the calibration corner point is a calibration point located at a corner point of the calibration area. Generating a calibration path based on the set positions of the calibration points includes: Determining, based on the first position, the second position, and the number of calibration points recorded in the set calibration point distribution information, a third position of the first calibration point other than the calibration center point and the calibration corner point and a fourth position of the rotation calibration point for visual calibration; According to the traversal order information recorded in the calibration point distribution information, a calibration path including the translation calibration point and the rotation calibration point is generated, and based on the set rotation angle, control information for controlling the rotation angle of the end effector at the rotation calibration point is generated, wherein the translation calibration point includes: the calibration center point, the calibration corner point and the first calibration point.

7. The method according to claim 5, characterized in that In response to the end effector moving to the position of the calibration point included in the calibration path, controlling the camera to perform image acquisition includes: If it is determined that the end effector moves to the position of the translation calibration point included in the calibration path, controlling the camera to perform image acquisition; If it is determined that the end effector moves to the position of the rotation calibration point included in the calibration path, the end effector is controlled to rotate a first angle and the camera is controlled to capture images, wherein the first angle is: the rotation angle indicated by the control information of the rotation calibration point.

8. The method according to claim 5, characterized in that Also includes: Obtaining recognition result evaluation information of each calibration image; Based on the recognition result evaluation information, the calibration path is adjusted and / or the visual processing parameters used when extracting feature points are adjusted.

9. The method according to any one of claims 1 to 8, characterized in that The first image processing operation linked list is recorded in a first image processing scheme, and the first image processing scheme is generated in the following manner: Obtain image processing flow editing information; If the image processing configuration mode indicated by the image processing flow editing information is scheme creation, generating a first image processing operation linked list based on the first image operation node and execution order information of the first image operation node included in the image processing flow editing information, and obtaining a first image processing scheme including the first image processing operation linked list; If the image processing configuration method indicated by the image processing flow editing information is scheme copying, then the second image processing operation linked list recorded in the second image processing scheme indicated by the image processing flow editing information is copied as the first image processing operation linked list to obtain the first image processing scheme including the first image processing operation linked list.

10. A target positioning device, characterized in that: The device comprises: An operator identification module, configured to identify an image processing operator for implementing the image processing operation represented by each first image operation node, wherein the first image operation node is an image operation node included in the first image processing operation linked list; An operator instantiation module is used to load the identified image processing operator from pre-packaged image processing operators, instantiate the loaded image processing operator, and obtain an instantiated operator; A calling sequence determining module, configured to determine the calling sequence of the instantiation operators according to the pointer directions between the first image operation nodes; A first position obtaining module is configured to sequentially call each instantiated operator to process an operator input image in the calling order, and obtain a first position of a task target in the image to be processed captured by the camera in the image coordinate system, wherein the operator input image of the first instantiated operator in the calling order is: the image to be processed, and the operator input images of the remaining instantiated operators except the first instantiated operator are: the image output after image processing by the forward adjacent instantiated operator of the instantiated operator; The second position determination module is used to determine the second position of the task target in the robot arm base coordinate system based on the first position and the calibration relationship, wherein the calibration relationship is: the transformation relationship between the image coordinate system and the robot arm base coordinate system.

11. The device according to claim 10, characterized in that Pre-configured with: a region of interest ROI and / or ROI inheritance information of a first image operation node; the first position acquisition module includes: A first ROI determination module is configured to use the first ROI as the target ROI of the instantiation operator if the ROI inheritance information configured on the first image operation node corresponding to the instantiation operator represents an inherited ROI, wherein the first ROI is: the ROI output by the first image operation node corresponding to the forward adjacent instantiation operator of the instantiation operator; a second ROI determination module is configured to use the second ROI as the target ROI of the instantiation operator if the first image operation node corresponding to the instantiation operator is not configured with ROI inheritance information or the configured ROI inheritance information represents a non-inherited ROI, wherein the second ROI is: the ROI configured by the first image operation node corresponding to the instantiation operator; a position acquisition submodule is configured to call each instantiation operator in the calling order, process the target ROI corresponding to each instantiation operator in the operator input image, and obtain the first position of the task target in the image to be processed captured by the camera in the image coordinate system; and / or Pre-configured with: ROI correction indication information of the first image operation node; and a second ROI determination module, specifically configured to, if the first image operation node corresponding to the instantiation operator is not configured with ROI inheritance information or the configured ROI inheritance information indicates that the ROI is not inherited, determine whether the ROI correction indication information configured by the first image operation node corresponding to the instantiation operator indicates a correction ROI; if not, use the second ROI as the target ROI of the instantiation operator, wherein the second ROI is: the ROI configured by the first image operation node corresponding to the instantiation operator; if yes, correct the second ROI based on the first ROI, and use the corrected ROI as the target ROI of the instantiation operator; and / or The instantiation operators include: a first type of operator and a second type of operator, wherein the first type of operator is an instantiation operator for implementing an image preprocessing operation, and the second type of operator is an instantiation operator for implementing a target positioning operation; the output of the first type of operator includes: an image obtained after image processing by the first type of operator and the first ROI; the output of the second type of operator includes: an image output by a forward adjacent instantiation operator and an ROI obtained after image processing by the second type of operator; and / or The calibration relationship is obtained according to the following modules: a calibration path generation module for generating a calibration path based on the positions of set calibration points; a calibration image acquisition module for controlling the movement of the end effector of the robotic arm according to the calibration path, and in response to the end effector moving to the position of the calibration point included in the calibration path, controlling the camera to perform image acquisition to obtain a calibration image; a calibration relationship acquisition module for determining that after the end effector traverses each calibration point included in the calibration path, based on the spatial position and image position of the feature point of the set marker in each captured calibration image, visually calibrate the robotic arm and the camera to obtain a calibration relationship, wherein the spatial position is determined based on the position of the calibration point included in the calibration path; and / or The positions of the calibration points include: a first position of a calibration center point and a second position of a calibration corner point, the calibration center point being a calibration point located at the center of a calibration area covered by a calibration path to be generated, and the calibration corner point being a calibration point located at a corner point of the calibration area, the calibration path generation module being specifically configured to determine, based on the first position, the second position, and the number of calibration points recorded in the set calibration point distribution information, a third position of a first calibration point and a fourth position of a rotation calibration point used for visual calibration in addition to the calibration center point and the calibration corner point; generate a calibration path including the translation calibration point and the rotation calibration point according to the traversal order information recorded in the calibration point distribution information, and generate control information for controlling the rotation angle of the end effector at the rotation calibration point based on the set rotation angle, wherein the translation calibration point includes: the calibration center point, the calibration corner point, and the first calibration point; and / or The calibration image acquisition module is specifically configured to control the camera to capture an image if it is determined that the end effector moves to the position of a translation calibration point included in the calibration path; and control the end effector to rotate a first angle and the camera to capture an image if it is determined that the end effector moves to the position of a rotation calibration point included in the calibration path, wherein the first angle is the rotation angle indicated by the control information of the rotation calibration point. and / or The first image processing operation linked list is recorded in a first image processing solution, and the first image processing solution is generated according to the following modules: An editing information acquisition module is used to obtain image processing flow editing information; a first scheme acquisition module is used to generate a first image processing operation linked list based on the first image operation node included in the image processing flow editing information and the execution order information of the first image operation node, if the image processing configuration mode indicated by the image processing flow editing information is scheme creation, and obtain a first image processing scheme including the first image processing operation linked list; a second scheme acquisition module is used to copy the second image processing operation linked list recorded in the second image processing scheme indicated by the image processing flow editing information as the first image processing operation linked list, if the image processing configuration mode indicated by the image processing flow editing information is scheme copying, and obtain a first image processing scheme including the first image processing operation linked list.

12. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 9 when executing a program stored in a memory.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images