Robotic vision servo control method for operating a biochemical instrument

Abstract

The application provides a robot vision servo control method for operating a biochemical instrument, which comprises: performing vision detection on a working area of the biochemical instrument, obtaining effective real-time images and hand-eye calibration parameters, extracting target contour features of a target to be operated, generating an initial vision pose and determining a near-field layout area; laying a disposable near-field reference body in a non-working position to obtain reference body contour features, a layout confirmation result and effective homographic images; generating local refraction observation data according to the effective homographic images, the reference body contour features, a specification identifier of the disposable near-field reference body and the hand-eye calibration parameters, solving a local refraction offset result and constructing a vision correction matrix; generating a real operation pose, an updated vision servo mapping relationship, a corrected motion trajectory, a current position reference sequence and an execution control queue according to the vision correction matrix; and performing dynamic vision detection to generate a current position detection sequence.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a visual servo control method for operating biochemical instruments. Background Technology

[0002] With the development of biochemical analyzers, nucleic acid amplification instruments, automated sample dispensers, and sample pretreatment workstations, using robots to grasp, align, insert, and place reagent tubes, reaction cups, well plates, caps, and micro-devices has become an important means to improve detection efficiency and reduce manual intervention. Current solutions typically acquire image information of the target object through a vision acquisition unit, then combine this with hand-eye calibration parameters to complete target localization, and control the end effector to perform the corresponding operation based on the localization result.

[0003] Most existing technologies rely on the relatively stable optical environment of the work area. However, biochemical instruments often have transparent enclosures, localized heating, localized cooling, moisture accumulation, and surface condensation, such as in constant-temperature reaction chambers, heated areas, cryogenic storage compartments, and behind protective windows. In these scenarios, the actual position of the target remains unchanged, but due to the combined effects of heat flow disturbances, condensation, and refraction through transparent media, the positions of openings, slots, covers, or edges acquired by the visual acquisition unit will experience localized visual offsets. This results in a discrepancy between the target position in the image and the actual reachable position of the robot.

[0004] Therefore, this invention proposes a robot vision servo control method for operating biochemical instruments. The information disclosed in the background section is only for enhancing understanding of the background of this disclosure and may therefore contain prior art information that is not common knowledge to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a robot vision servo control method for operating biochemical instruments, thereby solving the technical problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A robot vision servo control method for operating biochemical instruments includes the following steps:

[0008] S1. Perform visual inspection on the biochemical instrument operating area, acquire effective real-time images and hand-eye calibration parameters, extract the target contour features of the target to be operated, generate the initial visual pose, and determine the near-field deployment area.

[0009] S2. Control the end effector to place the near-field reference body in a non-operating position within the near-field placement area, confirm the placement status, and obtain the reference body contour features, placement confirmation results, and effective co-field image.

[0010] S3. Based on the effective co-field image, the contour features of the reference body, the specification markings of the one-time near-field reference body, and the hand-eye calibration parameters, generate local refraction observation data, solve for the local refraction offset results, and construct the visual correction matrix;

[0011] S4. Perform pose correction on the initial visual pose according to the visual correction matrix, generate the real operation pose and the updated visual servo mapping relationship, and generate the corrected motion trajectory, current position reference sequence and execution control queue according to the operation type.

[0012] S5. Control the end effector to execute the corrected motion trajectory according to the execution control queue, perform visual detection to generate the current position detection sequence, calculate the current position deviation, and output the matrix to update the trigger command or operation completion status according to the current position deviation and the current position deviation preset threshold. When the output matrix updates the trigger command, S2 to S4 are re-executed.

[0013] S1 specifically includes: performing visual inspection on the biochemical instrument operating area, continuously acquiring real-time images, filtering valid real-time images according to the integrity of the target outline, the occlusion ratio, and the reflective coverage ratio, and simultaneously reading the corresponding hand-eye calibration parameters; extracting the target outline features of the target based on the valid real-time images, and generating an initial visual pose including the target center position, the target principal axis direction, and the target operating surface normal direction in combination with the hand-eye calibration parameters; and performing local distortion discrimination visual inspection on the adjacent area of ​​the target based on the valid real-time images, target outline features, and initial visual pose, generating a near-field deployment area including the area center position, area boundary range, minimum spacing, and visible direction.

[0014] S2 specifically includes: reading the initial visual pose and near-field deployment area, controlling the end effector to deploy a disposable near-field reference body at a non-operational position near the target to be operated, and outputting the deployment status and the specification identifier of the disposable near-field reference body; performing deployment confirmation visual detection on the area where the target to be operated and the disposable near-field reference body are located, extracting the reference body contour features, and generating a deployment confirmation result; when the deployment confirmation result meets the requirements of co-field imaging, acquiring an image that simultaneously contains the target to be operated and the disposable near-field reference body, filtering out valid co-field images, and outputting the reference body contour features and the specification identifier of the disposable near-field reference body corresponding to the valid co-field images.

[0015] S3 specifically includes: reading the valid co-field image, the contour features of the reference body corresponding to the valid co-field image, the specification identifier of the one-time near-field reference body, and the hand-eye calibration parameters; calling the preset real geometric parameters corresponding to the specification identifier of the one-time near-field reference body; performing benchmark comparison visual inspection on the one-time near-field reference body; generating local refraction observation data including at least the lateral offset of the reference body center, the longitudinal offset of the reference body center, and the angular offset of the reference body contour; solving the local refraction offset result of the target to be operated on based on the local refraction observation data and the relative positional relationship between the target to be operated on and the one-time near-field reference body in the valid co-field image; and constructing a visual correction matrix for correcting the initial visual pose based on the local refraction offset result, the initial visual pose, and the hand-eye calibration parameters.

[0016] S4 specifically includes: reading the visual correction matrix, the initial visual pose, and the operation type corresponding to the target to be operated on; writing the visual correction matrix into the updated visual servo mapping relationship; performing pose correction on the initial visual pose to generate a true operation pose including the true center position, the true orientation, and the true operation surface normal; generating a corrected motion trajectory including an approach segment, an execution segment, and an exit segment based on the true operation pose, the updated visual servo mapping relationship, and the operation type; performing trajectory sampling processing on the corrected motion trajectory to generate a current position reference sequence including at least a reference time, a reference position, a reference orientation, and a trajectory segment identifier; and synchronously writing the current position reference sequence and the corrected motion trajectory into the execution control queue.

[0017] S5 specifically includes: reading the execution control queue, controlling the end effector to execute the approach segment, execution segment, and exit segment according to the corrected motion trajectory, and continuously performing dynamic visual detection to generate a current position detection sequence corresponding to the current position reference sequence; pairing the current position detection sequence with the current position reference sequence point by point, calculating the current position deviation, and outputting the operation continuation status, matrix update trigger command, or operation completion status based on the comparison result between the current position deviation and the current position deviation preset threshold; when the matrix update trigger command is output, suspending the current execution control queue and controlling the end effector to return to the safe waiting position, re-executing to generate the updated visual servo mapping relationship, corrected motion trajectory, current position reference sequence, and execution control queue; when the operation completion status is output, ending the operation of the current target to be operated on and outputting the operation result.

[0018] The beneficial effects of this invention are as follows:

[0019] This invention establishes a correction basis for local optical distortions in the current operation cycle by deploying a one-time near-field reference body in the vicinity of the target to be operated, and generating local refraction observation data and a visual correction matrix by combining effective co-field images, thereby improving the accuracy of determining the true operation pose of the target to be operated.

[0020] This invention directly writes the local refraction offset results into the updated visual servo mapping relationship, so that the robot end effector no longer relies on the uncorrected initial visual pose to perform operations, which can reduce the probability of insertion offset, grasping offset and alignment offset in transparent enclosures, thermal fields and condensation environments.

[0021] This invention divides the modified motion trajectory into an approach segment, an execution segment, and an exit segment, and generates corresponding trajectories in combination with the operation type. This enables the robot to maintain a consistent trajectory generation logic during insertion, grasping, alignment, and pick-up / placement, thereby improving the executability and stability of the control process.

[0022] This invention constructs a current position reference sequence and a current position detection sequence, and compares the current position deviation with a preset threshold for the current position deviation. This enables timely identification of trajectory misalignment during execution, preventing the end effector from continuing to operate along the misaligned trajectory.

[0023] This invention improves the stability of continuous operation of the robot in complex near-field environments by triggering a matrix update trigger command and re-executing the local refraction offset update process when the current position deviation exceeds a preset threshold. The updated visual servo mapping relationship and corrected motion trajectory replace the old results. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the robot vision servo control method for operating biochemical instruments according to the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example: Figure 1 As shown, this embodiment provides a robot vision servo control method for operating biochemical instruments, including the following steps:

[0027] S1. Perform visual inspection on the biochemical instrument operating area, acquire effective real-time images and hand-eye calibration parameters, extract the target contour features of the target to be operated, generate the initial visual pose, and determine the near-field deployment area.

[0028] S2. Control the end effector to place the near-field reference body in a non-operating position within the near-field placement area, confirm the placement status, and obtain the reference body contour features, placement confirmation results, and effective co-field image.

[0029] S3. Based on the effective co-field image, the contour features of the reference body, the specification markings of the one-time near-field reference body, and the hand-eye calibration parameters, generate local refraction observation data, solve for the local refraction offset results, and construct the visual correction matrix;

[0030] S4. Perform pose correction on the initial visual pose according to the visual correction matrix, generate the real operation pose and the updated visual servo mapping relationship, and generate the corrected motion trajectory, current position reference sequence and execution control queue according to the operation type.

[0031] S5. Control the end effector to execute the corrected motion trajectory according to the execution control queue, perform visual detection to generate the current position detection sequence, calculate the current position deviation, and output the matrix to update the trigger command or operation completion status according to the current position deviation and the current position deviation preset threshold. When the output matrix updates the trigger command, S2 to S4 are re-executed.

[0032] S1 specifically includes the following sub-steps:

[0033] S110: The control vision acquisition unit performs the first round of visual inspection on the biochemical instrument's operating area, continuously acquiring real-time images of the area where the target to be operated is located, and simultaneously reading the hand-eye calibration parameters corresponding to the current vision acquisition unit. The real-time images are preferably continuously acquired image sequences rather than single-frame images, so that image availability screening is completed first, and then images that meet the conditions are sent for subsequent processing.

[0034] During implementation, at least three items are checked for each frame of image: First, whether the target outline features of the target to be operated on are completely within the field of view; second, whether the end effector, door edge, or clamping component obstructs the main operating boundary of the target to be operated on; and third, whether the reflective area and condensation obstruction area of ​​the transparent cover cover the central area or main edge area of ​​the target to be operated on. Only when all three items meet the preset conditions is the corresponding image determined as a valid real-time image.

[0035] For example, a target contour coverage of no less than 90%, an occlusion ratio of no more than 10%, and a strong reflective coverage ratio of no more than 15% can be used as valid judgment criteria. The hand-eye calibration parameters include at least the transformation relationship from the visual acquisition unit coordinate system to the robot coordinate system, the image scale conversion relationship, and the distortion correction parameters. Their function is to uniformly map the image features obtained by subsequent visual detection to the spatial coordinates that the robot can access.

[0036] The output of S110 is not a generalized image result, but a filtered effective real-time image and the hand-eye calibration parameters corresponding to the effective real-time image, which are directly read by S120.

[0037] S120. Read the valid real-time image and hand-eye calibration parameters, perform a second round of visual detection on the target to be operated, extract the target contour features to characterize the operation boundary of the target to be operated, and generate an initial visual pose based on the target contour features. The target contour features are selected according to the type of the target to be operated: when the target to be operated is an opening, extract the closed edge of the opening and its center; when the target to be operated is a slot, extract the two sides of the slot, the bottom edge of the slot, and the end limiting edge; when the target to be operated is a cover, extract the annular edge or opening edge of the cover.

[0038] To avoid losing the basis for subsequent coordinate correction, the target contour features extracted in S120 should at least include the target center position features, the target principal axis direction features, and the target operation surface normal features. Then, substituting the target contour features into the coordinate transformation relationship corresponding to the hand-eye calibration parameters, the initial visual pose of the target to be operated in the robot coordinate system is generated, which can be expressed as:

[0039]

[0040] in, Indicates the initial visual pose; This indicates the transformation relationship from the vision acquisition unit coordinate system to the robot coordinate system; This represents the set of target contour features.

[0041] In this step, the initial visual pose includes at least the target center position, the target principal axis direction, and the target operation surface normal direction. Subsequently, S130 uses this initial visual pose as a reference to determine whether there is local distortion in the vicinity of the target to be operated. S210 then deploys a one-time near-field reference body based on the initial visual pose and the near-field deployment area. Therefore, the output of S120 is explicitly limited to the initial visual pose and target contour features, and no useless results are generated.

[0042] In terms of specific visual algorithm implementation, extracting target contour features can be achieved by using conventional edge detection algorithms based on the Canny or Sobel operators, combined with Hough transform to extract regular edges; or, in biochemical scenes with complex background reflections, image segmentation algorithms based on convolutional neural networks (such as the Mask R-CNN model) can be used to peel off the complex background, thereby accurately extracting closed edges and their centers.

[0043] S130: Read the effective real-time image, target contour features and initial visual pose, perform local distortion discrimination visual detection on the vicinity of the target to be operated, identify the image bending area, the area of ​​sudden change in brightness and darkness, the edge drift area and the condensation occlusion area, and generate the near field deployment area for deploying the one-time near field reference body.

[0044] In practice, taking the center position of the target to be operated on as a reference, a local detection window is established around it. The edge offset, brightness change, and occlusion ratio within the local detection window are calculated respectively, and then a comprehensive distortion index is constructed.

[0045]

[0046] in, This represents the overall distortion index; Indicates the edge offset; Indicates the amount of sudden change in brightness; Indicates the percentage of occlusion; , , These represent the weighting coefficients corresponding to edge offset, brightness abrupt change, and occlusion percentage, respectively.

[0047] In the embodiments, it is possible to take , , When the overall distortion index exceeds a preset distortion threshold, the local detection window is identified as a local distortion region; when the overall distortion index is not greater than the preset distortion threshold, the local detection window is identified as a candidate region for direct deployment. Subsequently, without obscuring the operational boundary of the target to be operated on and within the same near-field imaging range as the target, a near-field deployment region is determined from the candidate regions for direct deployment. The near-field deployment region outputs at least four fields: the region center position, the region boundary range, the minimum distance from the operational boundary of the target to be operated on, and the visible direction allowing a single near-field reference body to enter.

[0048] For example, the minimum spacing can be set to 2 to 5 millimeters to ensure that the one-time near-field reference body is close enough to the target to be operated on to reflect the local refraction offset, but does not contact the target to be operated on and affect subsequent operations.

[0049] The output of S130 is the near-field deployment area, which is directly used in S210 as the spatial basis for the one-time near-field reference body to enter the non-operational position.

[0050] S2 specifically includes the following sub-steps:

[0051] S210: Read the initial visual pose output by S120 and the near-field deployment area output by S130. According to the current spatial position of the target to be operated, the visible direction that allows the one-time near-field reference body to enter, and the area boundary range of the near-field deployment area, control the end effector to carry the one-time near-field reference body into the near-field deployment area along the preset approach path, and position the one-time near-field reference body at a non-operational position near the target to be operated.

[0052] The "disposable near-field reference body" in this step refers to a reference component that participates in the local refraction offset measurement within a single operation cycle and is no longer reused as the next target to be operated after the measurement is completed. Its outer contour, size and edge shape have been pre-calibrated at the factory and a corresponding relationship has been established with the unique disposable near-field reference body specification identifier so that S310 can call the corresponding preset real geometric parameters according to the specification identifier.

[0053] In this step, the "non-operational position" refers to a position where the near-field reference body is within the same near-field imaging range as the target to be operated on, can enter the field of view of the visual acquisition unit simultaneously with the target, and does not contact the target or obstruct its main operational boundaries. To avoid the non-operational position remaining merely a functional description, this embodiment provides spatial determination rules for the non-operational position, which can be expressed as:

[0054]

[0055] in, This represents the center distance between the center of the near-field reference body and the center of the target to be operated on; Indicates the center position of a one-time near-field reference body; Indicates the center position of the target to be operated on; Indicates the preset minimum spacing; Indicates the preset maximum spacing; This indicates the projection occlusion ratio of the primary operational boundary of the target to be operated on by a single near-field reference body; This indicates the maximum allowable occlusion percentage.

[0056] During implementation, it can be Set to 2 millimeters, Set to 5 millimeters, Setting it to 10%, the one-time near-field reference body is close enough to the target to be operated on, so as to truly reflect the local refraction shift in the vicinity of the target to be operated on, without obscuring or contacting the target to be operated on due to being too close.

[0057] After the end effector completes the one-time near-field reference body deployment, it does not keep the clamping component permanently between the target to be operated and the one-time near-field reference body. Instead, it exits the current main field of view along a preset retreat direction to prevent the end effector itself from entering the subsequent deployment confirmation image. The output of S210 is the deployment status of the one-time near-field reference body and the specification identifier of the one-time near-field reference body. The deployment status includes at least the current center position, current orientation, and retreat completion status, which are available for S220 to call.

[0058] In one specific implementation scenario, the physical carrier of the disposable near-field reference body can be a heat-resistant plastic block or patch with a high-contrast QR code, standard checkerboard pattern or specific geometric positioning mark printed on its surface; its shape and surface friction are adapted to the gripper of the end effector, so that the end effector can achieve stable picking and fixed-point placement of the disposable near-field reference body through a general gripping action similar to gripping a conventional biochemical test tube.

[0059] S220: Read the deployment status and the specification mark of the one-time near-field reference body, control the vision acquisition unit to perform deployment confirmation visual detection on the area where the target to be operated and the one-time near-field reference body are located, extract the reference body contour features of the one-time near-field reference body, and check the relative distance, relative orientation and image occlusion status between the one-time near-field reference body and the target to be operated, and generate deployment confirmation results.

[0060] The "reference body contour features" in this step refer to a set of image features that can uniquely characterize the geometric state of the outer contour of a one-time near-field reference body. These features include at least one of the following: reference body edge position, reference body corner position, or reference body closed contour center. This ensures that subsequent step S310 can compare the current image contour with preset true geometric parameters. To ensure that the deployment confirmation result has a clear basis for judgment, rather than relying solely on human experience, this embodiment sets a quality scoring rule for the deployment confirmation process, which can be expressed as:

[0061]

[0062] in, Indicates the deployment confirmation quality score; Indicates the completeness of the near-field reference body profile in a single operation; This indicates the integrity of the main operational boundaries of the target to be operated on; Indicates the occlusion ratio in the current image; , , These represent the weighting coefficients corresponding to the integrity of the reference body contour, the integrity of the target boundary, and the occlusion ratio, respectively.

[0063] When implementing, it is possible to take , And stipulate when When the value is not lower than 0.85, the current deployment status is determined to meet the requirements for co-field imaging; when... If the value is below 0.85, the current deployment status is determined to not meet the requirements for co-field imaging. If the deployment confirmation result indicates that the requirements for co-field imaging are not met, the control end effector performs fine-tuning according to the visible direction in the near-field deployment area where a single near-field reference body can enter. The fine-tuning amount can be adjusted step by step in increments of 0.2 mm to 0.5 mm, and visual inspection is repeated after each fine-tuning until a deployment confirmation result that meets the requirements for co-field imaging is obtained.

[0064] For example, if a disposable near-field reference body is located within the near-field deployment area, but its right edge obscures the upper edge of the target aperture, the disposable near-field reference body can be slightly moved by 0.3 mm in a tangential direction away from the upper edge of the aperture, and then the deployment confirmation visual inspection can be performed again. The output of S220 includes the reference body contour features, deployment confirmation results, and disposable near-field reference body specification markings, which can be used by S230 and S310.

[0065] S230. When the deployment confirmation results indicate that the current deployment status meets the requirements for co-field imaging, the visual acquisition unit is controlled to perform co-field visual detection on the target to be operated and the disposable near-field reference body, continuously acquiring images that simultaneously contain the target to be operated and the disposable near-field reference body, and selecting valid co-field images from the continuously acquired images. The "valid co-field image" in this step refers to an image that can completely represent the main operational boundaries of the target to be operated, completely represent the contour features of the disposable near-field reference body, and is consistent with the current deployment confirmation results.

[0066] To ensure that the S310 subsequent solution of local refraction observation data uses input with stable quality and clear parameter sources, this embodiment stipulates that at least three consecutively acquired images are used, and valid co-field images are selected according to the following rules: First, the integrity of the contour of the one-time near-field reference body reaches a preset threshold; second, the integrity of the main operational boundary of the target to be operated on reaches a preset threshold; third, there are no new occlusions or obvious out-of-focus areas in the image; fourth, the relative positional relationship between the target to be operated on and the one-time near-field reference body is consistent with the relative distance and relative orientation in the deployment confirmation results. If multiple images simultaneously meet the above conditions, the frame with the highest clarity and the smallest brightness fluctuation is selected as the valid co-field image.

[0067] For example, in three consecutively acquired images, if the first frame has slight reflective stripes, the second frame has a complete outline and is unobstructed, and the third frame has slight defocus, then the second frame is selected as the valid co-field image. To ensure the input closed loop of S310, S230 not only outputs the valid co-field image but also simultaneously outputs the corresponding reference body contour features and one-time near-field reference body specification identifier, and establishes a correlation record among the three. This allows S310 to directly retrieve the reference body contour features and one-time near-field reference body specification identifier corresponding to the same image when reading the valid co-field image. Combined with the hand-eye calibration parameters output by S110 and the preset true geometric parameters obtained from the specification identifier index, subsequent local refraction observation data generation is performed. The output of S230 consists of the valid co-field image, the reference body contour features corresponding to the valid co-field image, and the one-time near-field reference body specification identifier, which are available for S310 to use.

[0068] S3 specifically includes the following sub-steps:

[0069] S310 reads the valid co-field image output by S230, the corresponding reference body contour features, and the disposable near-field reference body specification identifier. It then calls the hand-eye calibration parameters output by S110 to perform a benchmark comparison visual inspection on the disposable near-field reference body, generating local refraction observation data. The "preset true geometric parameters" in this step refer to a set of standard geometric data pre-written during the factory calibration stage of the disposable near-field reference body and uniquely indexed by the disposable near-field reference body specification identifier. This data includes at least the reference body center position, reference body edge length, relative positions of reference body corners, and reference body reference orientation. The "benchmark comparison visual inspection" in this step refers to extracting the current imaging contour of the disposable near-field reference body from the valid co-field image and comparing this current imaging contour with the standard contour corresponding to the preset true geometric parameters item by item to obtain the offset of the imaging coordinates caused by the current near-field environment.

[0070] In implementation, the current contour center, current edge position, and current contour orientation of the one-time near-field reference body are first extracted based on the contour features of the reference body. Then, a unified mapping between image coordinates and robot coordinates is completed through hand-eye calibration parameters. Subsequently, the difference is calculated with the preset real geometric parameters. To avoid local refraction observation data being insufficient to support subsequent solutions due to only a single deviation value, this embodiment explicitly limits the local refraction observation data to at least including the lateral offset of the reference body center, the longitudinal offset of the reference body center, the angular offset of the reference body contour, and the bending amount of the reference body edge.

[0071] Among them, the lateral offset and longitudinal offset of the reference body center are used to characterize the translational effect of local refraction on the position coordinates, the angular offset of the reference body contour is used to characterize the rotational effect of local refraction on the orientation judgment, and the bending amount of the reference body edge is used to characterize the non-rigid distortion effect of local refraction on the edge shape.

[0072] For example, when the center of a single near-field reference body extracted from an effective co-field image is offset by 0.18 mm horizontally, 0.11 mm vertically, and 1.6 degrees in profile orientation relative to its corresponding center with preset true geometric parameters, the system will write these three offsets, along with the corresponding edge bending amount, into the local refraction observation data, instead of retaining only a single composite value. The output of S310 is the local refraction observation data, which is then called by S320.

[0073] S320: Read the local refraction observation data and the current imaging position of the target to be operated on in the effective co-field image output by S230, as well as the relative positional relationship between the target to be operated on and the one-time near-field reference body within the effective co-field image. Solve for the local refraction offset result in the near-field region where the target to be operated is located. The "relative positional relationship" in this step refers to the center distance, azimuth angle, and depth-of-field relationship between the target to be operated on and the one-time near-field reference body in the same effective co-field image. Since the one-time near-field reference body and the target to be operated on are positioned within the same near-field imaging range, the local refraction effects they experience are of the same origin. Therefore, the coordinate correction amount of the target to be operated on can be estimated using the local refraction observation data of the one-time near-field reference body.

[0074] To make this estimation process feasible, this embodiment records the local refraction observation data of a one-time near-field reference body as the reference body observation vector, and records the local refraction offset result of the target to be operated on as the target correction vector. The relationship between the two can be expressed as:

[0075]

[0076] in, This represents the target correction vector of the target to be operated on; This represents the mapping coefficient matrix determined by the relative positional relationship between the target to be manipulated and a one-time near-field reference body; The reference volume observation vector represents a one-time near-field reference volume.

[0077] The "reference body observation vector" in this step consists of the reference body center lateral offset, reference body center longitudinal offset, and reference body contour angle offset output by S310; the "target correction vector" in this step includes at least the target center lateral correction, target center longitudinal correction, and target orientation correction. Mapping coefficient matrix Its function is to allocate the offset measured at the current position of the one-time near-field reference body to the target to be operated on according to the relative positional relationship between the target to be operated and the one-time near-field reference body.

[0078] During implementation, a mapping coefficient table for different center distances and azimuth angles can be pre-established based on the installation calibration results, and then called upon in this step according to the relative positional relationship recorded in the currently valid co-field image; alternatively, during the initial deployment of the equipment, a one-time co-view calibration of the near-field reference body and the target to be operated can be completed using multiple sets of standard samples to obtain the mapping coefficient matrix. The range of values ​​for .

[0079] For example, when the center distance between the target to be manipulated and the one-time near-field reference body is 3.2 mm, the azimuth angle is 18 degrees, and both are at the same depth of field level, the corresponding mapping coefficient can be selected from the mapping coefficient table. If the reference body observation vector at this time The data shows that the lateral offset of the reference body center is 0.18 mm, the longitudinal offset of the reference body center is 0.11 mm, and the angular offset of the reference body contour is 1.6 degrees. After mapping, the lateral correction of the target center, the longitudinal correction of the target center, and the target orientation correction of the target to be operated can be obtained.

[0080] Therefore, S320 outputs not an abstract judgment, but a set of definite correction values ​​that can be directly used in subsequent matrix construction. The output of S320 is the local refraction offset result, which is used by S330.

[0081] The specific common-view calibration process is as follows: Under ideal conditions of no refraction, room temperature, and no obstruction, the standard relative position of the reference body and the target to be operated is visually detected and recorded; then, under the working conditions of introducing a transparent cover, heating a thermal field, or cooling and condensing to generate an actual local refraction environment in the biochemical instrument, the current distortion coordinates of the two in the same field of view are repeatedly collected; by comparing multiple sets of coordinate offsets under different spacings and angles, the corresponding spatial offset mapping weights are calculated using multiple linear regression or nonlinear fitting algorithms, thereby constructing a mapping coefficient table.

[0082] S330: Read the local refraction offset results, initial visual pose, and hand-eye calibration parameters; construct a visual correction matrix from image distortion correction to robot spatial pose correction; and output the visual correction matrix. The "visual correction matrix" in this step refers to the pose transformation matrix used to correct the initial visual pose to the actual operational pose. Essentially, within the current work cycle, it combines the target center lateral correction, target center longitudinal correction, and target orientation correction output by S320 according to the coordinate form required for robot control.

[0083] Considering that this invention focuses on addressing the impact of local optical distortion on position and orientation in the near-field environment, this embodiment employs a planar pose correction method to construct the visual correction matrix, which can be expressed as:

[0084]

[0085] in, Represents the visual correction matrix; This indicates the lateral correction amount for the target center; This indicates the longitudinal correction amount for the target center; This indicates the target orientation correction amount.

[0086] In this step, the correction values ​​in the local refraction offset results are first uniformly converted to a robot coordinate expression form consistent with the initial visual pose, and then the visual correction matrix is ​​constructed according to the above formula. This matrix is ​​not an independently stored mathematical result, but is used to directly correct the initial visual pose, so that when the S410 updates the visual servo mapping relationship, it can redefine the position and orientation of the target to be operated based on the actual offset state of the current near field environment.

[0087] For example, when the target center lateral correction amount output by S320 is 0.16 mm, the target center longitudinal correction amount is 0.09 mm, and the target orientation correction amount is 1.4 degrees, S330 constructs the visual correction matrix corresponding to the current work cycle according to the above values, and establishes a binding relationship between the visual correction matrix and the current work cycle identifier to prevent subsequent steps from mistakenly calling the matrix of the previous work cycle.

[0088] The output of S330 is a visual correction matrix, which can be directly called by S410 to correct the initial visual pose to the actual operational pose.

[0089] S4 specifically includes the following sub-steps:

[0090] S410: Read the visual correction matrix output by S330, and read the initial visual pose output by S120. At the same time, read the operation type corresponding to the target to be operated, write the visual correction matrix into the updated visual servo mapping relationship corresponding to the current job cycle, perform pose correction on the initial visual pose, and generate the real operation pose of the target to be operated.

[0091] The "updated visual servo mapping relationship" in this step refers to the set of mapping parameters for the current job cycle called when converting visual detection results into robot end-effector control quantities. This includes at least the mapping parameters from image coordinates to robot coordinates, target pose correction parameters, and trajectory planning call identifiers. The "real operating pose" in this step refers to the true center position, true orientation, and true operating surface normal of the target in the robot coordinate system, assuming that local optical distortion in the current biochemical instrument operating area has been eliminated. To avoid the "pose correction" remaining merely a result description, this embodiment expresses the generation rules for the real operating pose as follows:

[0092]

[0093] in, Indicates the actual operating pose; Represents the visual correction matrix; This indicates the initial visual pose.

[0094] In this step, the visual correction matrix corresponding to the current job cycle is first called to correct the target center position and target orientation in the initial visual pose. Then, the corrected pose result is written into the updated visual servo mapping relationship so that subsequent trajectory planning will no longer call the uncorrected initial visual pose.

[0095] For example, when the visual correction matrix output by S330 corresponds to a target center lateral correction of 0.16 mm, a target center longitudinal correction of 0.09 mm, and a target orientation correction of 1.4 degrees, S410 directly applies this matrix to the initial visual pose to obtain the true center position and true orientation of the target to be operated on. This result is then bound and recorded with the current work cycle identifier to prevent subsequent steps from mistakenly calling the pose correction result from the previous work cycle. The output of S410 is the true operation pose, the updated visual servo mapping relationship, and the operation type, which are then called by S420.

[0096] S420: Read the actual operation pose, the updated visual servo mapping relationship, and the operation type, and generate a corrected motion trajectory for the end effector to reach the target. The "operation type" in this step includes at least one of insertion, grasping, alignment, and pick-and-place; the "corrected motion trajectory" in this step refers to the continuous control trajectory under the constraints of the actual operation pose, transitioning from a safe waiting position to the vicinity of the target, completing the target operation, and exiting the current area.

[0097] To ensure the feasibility of trajectory generation, this embodiment divides the corrected motion trajectory into an approach segment, an execution segment, and an exit segment. The approach segment moves the end effector from a safe waiting position to a preparatory position outside the actual operating pose, ensuring the end effector does not enter the main operating boundary of the target during the approach. The execution segment completes the actual operation based on the actual operating pose. When the operation type is insertion, the execution segment enters along the normal to the actual operating surface. When the operation type is grasping, the execution segment uses the actual center position as the clamping center and aligns the clamping component according to the actual orientation before closing. When the operation type is alignment, the execution segment constrains the posture matching with both the actual orientation and the normal to the actual operating surface. When the operation type is pick-up or placement, the execution segment completes the extraction or placement under the combined constraints of the actual center position and the actual orientation. The exit segment, after the execution segment ends, causes the end effector to leave the vicinity of the target along a preset exit direction, avoiding lingering in areas with significant local distortion and affecting the next visual inspection.

[0098] During implementation, the corrected motion trajectory includes at least the trajectory segment identifier, the target position of each trajectory segment, the target orientation of each trajectory segment, and the execution order. For example, when the operation type is insertion, the end effector can first be moved 3 mm above the target to be operated as the approach segment endpoint, then the insertion is completed along the normal of the actual operation surface as the execution segment, and after the insertion is completed, it is withdrawn 4 mm in the opposite direction to the insertion direction as the exit segment. The output of S420 is the corrected motion trajectory, which is called by S430 and S510.

[0099] S430: Read the corrected motion trajectory, perform trajectory sampling processing on the corrected motion trajectory, generate a current position reference sequence that corresponds one-to-one with the corrected motion trajectory, and write the current position reference sequence and the corrected motion trajectory into the execution control queue synchronously.

[0100] The "current position reference sequence" in this step refers to a set of reference sampling points arranged in the execution order of the corrected motion trajectory. Each reference sampling point includes at least a reference time, reference position, reference orientation, and the identifier of the trajectory segment it belongs to, so that it can be used by S510 to obtain the current position detection sequence and by S520 to calculate the current position deviation point by point. To avoid the "trajectory sampling processing" remaining an abstract description, this embodiment defines the sampling generation relationship as follows:

[0101]

[0102] in, Indicates the first One reference sampling point; Indicates the correction of the first on the trajectory. One sampling location; This represents the sampling generation relationship that converts the sampling location into a reference time, reference position, and reference orientation; Indicates the sampling sequence number.

[0103] During implementation, sampling can be performed at time intervals or at path length intervals; for example, the corrected motion trajectory can be sampled point by point using a 10-millisecond time interval or a 0.2-millimeter path length interval. If time interval sampling is used, the reference position and reference orientation are extracted for the trajectory position at each moment; if path length interval sampling is used, the reference position and reference orientation are extracted for each position where the cumulative path length reaches the preset sampling step size.

[0104] To ensure a strict correspondence in subsequent deviation determinations, this step arranges all reference sampling points in the execution order to form a current position reference sequence, and writes them into the execution control queue in the same order, so that the current position detection sequence output by S510 can be paired one-to-one with the current position reference sequence according to the sampling number or reference time.

[0105] For example, when the total length of the approach segment is 4 mm and the sampling step size is 0.2 mm, 20 reference sampling points can be generated; when the execution segment length is 3 mm, 15 more reference sampling points can be generated; when the exit segment length is 4 mm, another 20 reference sampling points are generated, and finally, all the reference sampling points form a complete current position reference sequence. The output of S430 is the current position reference sequence and the execution control queue, wherein the current position reference sequence is called by S520, and the execution control queue is called by S510.

[0106] S5 specifically includes the following sub-steps:

[0107] S510 reads the execution control queue output by S430 and calls the updated visual servo mapping relationship corresponding to the current job cycle. It controls the end effector to execute the approach segment, execution segment and exit segment in sequence according to the corrected motion trajectory. At the same time, it continuously performs dynamic visual detection on the target to be operated, the near field area where the one-time near field reference body is located and the current position of the end effector, and generates the current position detection sequence corresponding to the current position reference sequence with the same sampling rhythm.

[0108] The “current position detection sequence” in this step refers to the set of current position data collected during the process of the end effector performing the correction motion trajectory, according to the sampling time or the sampling sequence number that is consistent with the current position reference sequence. Each current position detection point includes at least the detection time, detection position, detection orientation and the identifier of the trajectory segment to which it belongs.

[0109] To ensure a one-to-one correspondence in subsequent comparisons of current position deviations, this step does not use arbitrary sampling rhythms independent of the current position reference sequence, but directly calls the sampling rules of the current position reference sequence; when the current position reference sequence is generated at 10-millisecond intervals, the current position detection sequence is also generated at 10-millisecond intervals; when the current position reference sequence is generated at 0.2-millimeter path length intervals, the current position detection sequence is also generated at 0.2-millimeter path length intervals.

[0110] During dynamic visual inspection, an image is first acquired of the end effector's current position. Then, the detection position and orientation of the end effector relative to the target are extracted from the current image. Finally, the detection result is written into the current position detection sequence. If the detection position cannot be directly extracted from the current image due to partial occlusion, short-term reflection enhancement, or momentary defocus, the previous valid detection point and the updated visual servo mapping relationship corresponding to the current job cycle are called to perform a compensation estimation on the current position. The compensation estimation result is then written into the current detection point of the current position detection sequence, ensuring that the subsequent point-by-point comparisons of S520 are uninterrupted.

[0111] For example, in the execution segment of the insertion operation, if the current position cannot be stably identified at the 12th detection time due to short-term reflection at the edge of the end effector, the detection position and orientation of the 11th valid detection point and the reference position corresponding to the 12th reference sampling point are called to perform a current position compensation estimation, and then the estimated detection position and orientation are written to the 12th detection point. The output of S510 is the current position detection sequence, which is called by S520.

[0112] S520. Read the current position reference sequence and the current position detection sequence, and according to the sampling sequence number or reference time, select the first... The detection point and the current location reference sequence in the _ ... Each reference sampling point is paired one by one, the current position deviation is calculated point by point, and the current position deviation is compared with the current position deviation preset threshold to generate an operation continue status, matrix update trigger command or operation complete status.

[0113] In this step, "current position deviation" refers to the combined deviation value consisting of the positional deviation between the detection position and the reference position, and the orientation deviation between the detection orientation and the reference orientation, under the same sampling sequence number. To clarify the generation rule for the current position deviation, this embodiment will... The current position deviation is represented as:

[0114]

[0115] in, Indicates the first Current position deviation; Indicates the first One detection location; Indicates the first One reference position; Indicates the first One detection orientation; Indicates the first One reference orientation; This represents the position deviation weighting coefficient; This represents the orientation deviation weighting coefficient.

[0116] When implementing, it is possible to take , This is to ensure that position deviation has a higher weight in the overall deviation, while orientation deviation participates in the determination as an attitude correction constraint. Then, the first... The current position deviation and the current position deviation preset threshold Compare and generate an update trigger flag:

[0117]

[0118] in, Indicates the first The update trigger flag corresponding to each detection point; This indicates the current position deviation preset threshold.

[0119] When the update trigger flag corresponding to any detection point When the value is 1, a matrix update trigger instruction is generated immediately; when the update trigger flags of the currently compared detection points are all equal to 0, and there are still unexecuted control items in the execution control queue, the operation continues status is output; when the last detection point completes the comparison and the end effector has completed the operation of the current target to be operated according to the execution control queue, the operation complete status is output.

[0120] For example, during a pick-up and place operation, if the position deviation of the 8th detection point is 0.12 mm and the orientation deviation is 0.6 degrees, and the current position deviation calculated according to the above weights does not exceed the preset threshold for the current position deviation, then the output operation continues. If the position deviation of the 14th detection point increases to 0.42 mm, and the combined deviation exceeds the preset threshold for the current position deviation, then a matrix update trigger command is immediately output. The output of S520 is either the operation continues status, the matrix update trigger command, or the operation complete status, which can be called by S530.

[0121] S530: Read the operation continuation status, matrix update trigger instruction, or operation completion status; when the operation continuation status is read, keep the current execution control queue running, and do not replace the visual correction matrix, corrected motion trajectory, current position reference sequence, and updated visual servo mapping relationship of the current job cycle; when the matrix update trigger instruction is read, immediately stop the remaining control items in the current execution control queue that have not yet been executed, and freeze the old corrected motion trajectory, old current position reference sequence, old execution control queue, and old updated visual servo mapping relationship corresponding to the current job cycle, and then control the end effector to return to the safe waiting position.

[0122] The "safe waiting position" in this step refers to the position where the end effector has exited the vicinity of the target to be operated on, will not obstruct subsequent in-field visual detection, and allows the near-field reference body to re-enter the near-field deployment area in one go. After the end effector reaches the safe waiting position, it re-enters the local refraction offset update process, that is, it re-executes S210 to S430, sequentially generating the updated reference body contour features, updated effective in-field image, updated local refraction observation data, updated local refraction offset results, updated visual correction matrix, updated true operating pose, updated visual servo mapping relationship, updated corrected motion trajectory, updated current position reference sequence, and updated execution control queue. The frozen old updated visual servo mapping relationship, old corrected motion trajectory, old current position reference sequence, and old execution control queue are replaced with the updated visual servo mapping relationship, updated corrected motion trajectory, updated current position reference sequence, and updated execution control queue. Then, the end effector is controlled to continue executing the updated execution control queue from the safe waiting position.

[0123] The purpose of this processing is to prevent the use of the old, inaccurate trajectory when a new local optical distortion occurs in the current near-field environment or when the current position deviation exceeds the preset threshold. Instead of continuing to execute the operation, the correction result regenerated in the current work cycle is used to take over the subsequent actions.

[0124] For example, in a grasping operation, if the end effector approaches the target and the current position deviation exceeds the threshold due to enhanced local condensation, the old execution control queue is terminated first, then the device returns to the safe waiting position, and S210 to S430 are re-executed to generate the updated visual correction matrix, the updated visual servo mapping relationship, and the updated corrected motion trajectory, and then the updated grasping action is continued.

[0125] When the operation completion status is read, the control end effector ends the operation of the current target to be operated and outputs the operation result. The operation result includes at least the current job cycle identifier, operation type, completion status and whether a matrix update has occurred, so that it can be recorded and recalled after the operation of the current target to be operated is completed.

[0126] All the above formulas are performed using dimensionless numerical calculations; the relevant formulas are based on empirical models that approximate the real situation, obtained through extensive data collection and software simulation fitting. The preset parameters and thresholds involved in the formulas can be conventionally set and adjusted by those skilled in the art according to the physical constraints of the actual application scenario.

[0127] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0128] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0129] In conclusion, the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A robot vision servo control method for operating biochemical instruments, characterized in that, Includes the following steps: S1. Perform visual inspection on the biochemical instrument operating area, acquire effective real-time images and hand-eye calibration parameters, extract the target contour features of the target to be operated, generate the initial visual pose, and determine the near-field deployment area, specifically including: Based on the effective real-time image, target contour features and initial visual pose, perform local distortion discrimination visual detection on the vicinity of the target to be operated, and generate a near-field deployment area including the center position of the region, the boundary range of the region, the minimum spacing and the viewing direction; The near-field deployment area should output at least four fields: the center position of the area, the range of the area boundary, the minimum distance between the area and the boundary of the target to be operated, and the visible direction that allows a one-time near-field reference body to enter. S2. The control end effector deploys a near-field reference body at a non-operational position within the near-field deployment area, confirms the deployment status, and obtains the reference body contour features, deployment confirmation results, and effective co-field images; specifically including: Read the initial visual pose and near-field deployment area, control the end effector to deploy a one-time near-field reference body in a non-operational position near the target to be operated, and output the deployment status and one-time near-field reference body specification identifier; Perform deployment confirmation visual inspection on the area where the target object and the one-time near-field reference body are located, extract the contour features of the reference body, and generate deployment confirmation results; When the deployment confirmation results meet the requirements for co-field imaging, images containing both the target to be operated and a disposable near-field reference body are acquired, valid co-field images are selected, and the reference body contour features and disposable near-field reference body specification identifiers corresponding to the valid co-field images are output. A disposable near-field reference body refers to a reference component that participates in local refraction and offset measurement within a single operation cycle and is not reused as the next target to be operated on after the measurement is completed. The non-operational position refers to the position where the near-field reference body is within the same near-field imaging range as the target to be operated, can enter the field of view of the visual acquisition unit at the same time as the target to be operated, and does not contact the target to be operated or obstruct the main operation boundary of the target to be operated; An effective co-field image refers to an image that can completely represent the main operational boundaries of the target to be operated on, as well as the contour features of the one-time near-field reference body, and is consistent with the current deployment confirmation results; S3. Based on the valid co-field image, the contour features of the reference body, the specifications of the one-time near-field reference body, and the hand-eye calibration parameters, generate local refraction observation data, solve for the local refraction offset results, and construct a visual correction matrix, specifically including: Read the valid co-field image, the contour features of the reference body corresponding to the valid co-field image, the specification mark of the one-time near-field reference body, and the hand-eye calibration parameters. Call the preset real geometric parameters corresponding to the specification mark of the one-time near-field reference body, perform benchmark comparison visual inspection on the one-time near-field reference body, and generate local refraction observation data including at least the lateral offset of the reference body center, the longitudinal offset of the reference body center, and the angular offset of the reference body contour. Based on local refraction observation data and the relative positional relationship between the target to be operated and the one-time near-field reference body in the effective co-field image, the local refraction offset result of the target to be operated is solved; based on the local refraction offset result, the initial visual pose and hand-eye calibration parameters, a visual correction matrix is ​​constructed to correct the initial visual pose. S4. Perform pose correction on the initial visual pose according to the visual correction matrix, generate the real operation pose and the updated visual servo mapping relationship, and generate the corrected motion trajectory, current position reference sequence and execution control queue according to the operation type.

2. The robot vision servo control method for operating biochemical instruments according to claim 1, characterized in that, Also includes: S5. Control the end effector to execute the corrected motion trajectory according to the execution control queue, perform visual detection to generate the current position detection sequence, calculate the current position deviation, and output the matrix to update the trigger command or operation completion status according to the current position deviation and the current position deviation preset threshold. When the output matrix updates the trigger command, S2 to S4 are re-executed.

3. The robot vision servo control method for operating biochemical instruments according to claim 1, characterized in that, S1 specifically includes: Visual inspection is performed on the biochemical instrument operation area, and real-time images are continuously acquired. Valid real-time images are selected according to the integrity of the target outline, the occlusion ratio, and the reflective coverage ratio. At the same time, the corresponding hand-eye calibration parameters are read. Extract the target contour features of the target to be operated from the effective real-time image, and generate an initial visual pose including the target center position, the target principal axis direction and the target operation surface normal direction by combining the hand-eye calibration parameters.

4. The robot vision servo control method for operating biochemical instruments according to claim 1, characterized in that, S4 specifically includes: Read the visual correction matrix, initial visual pose and operation type corresponding to the target to be operated, write the visual correction matrix into the updated visual servo mapping relationship, perform pose correction on the initial visual pose, and generate a real operation pose including the real center position, real orientation and real operation surface normal. Based on the actual operation pose, the updated visual servo mapping relationship, and the operation type, a corrected motion trajectory including an approach segment, an execution segment, and an exit segment is generated; Perform trajectory sampling processing on the corrected motion trajectory to generate a current position reference sequence that includes at least the reference time, reference position, reference orientation, and trajectory segment identifier. Then, write the current position reference sequence and the corrected motion trajectory into the execution control queue in sync.

5. The robot vision servo control method for operating biochemical instruments according to claim 2, characterized in that, S5 specifically includes: The execution control queue is read, and the end effector is controlled to execute the approach segment, execution segment and exit segment according to the corrected motion trajectory. Dynamic visual detection is continuously performed to generate a current position detection sequence corresponding to the current position reference sequence. The current position detection sequence is paired point by point with the current position reference sequence, the current position deviation is calculated, and the operation continuation status, matrix update trigger command or operation completion status is output based on the comparison result between the current position deviation and the current position deviation preset threshold. When the output matrix update triggers an instruction, the current execution control queue is aborted and the end effector is controlled to return to a safe waiting position.

6. The robot vision servo control method for operating biochemical instruments according to claim 5, characterized in that, Also includes: Re-execute to generate updated visual servo mapping, correct motion trajectory, current position reference sequence, and execution control queue. When the operation completion status is output, end the operation on the current target to be operated and output the operation result.

Images