Method for calibrating an optical recording device

Abstract

By at least using a model panel (140) and a first calibration view for calibrating (216) an optical recording device (150) associated with a field of view and an image plane, the model panel (140) comprising a set of calibration marks (40-49), wherein the first calibration view comprises a first set of calibration positions (20-29) of the field of view, wherein the method comprises at least the steps of: constructing (204, 207) at least a plurality of scenes such that for each position of the first set of calibration positions (20-29) a calibration mark of the set of marks is located in a respective scene of the plurality of scenes at said each position, and wherein each scene of the plurality of scenes is constructed by translating the model panel (140) to a respective position relative to the optical recording device (150); acquiring (205, 208) at least a plurality of images (920, 930) such that each scene of the plurality of scenes is displayed in a respective image of the plurality of images (920, 930); locating (206, 209) at least a plurality of calibration sites (60-69) such that for each position of the first set of calibration positions (20-29) a respective site of the plurality of calibration sites (60-69) is comprised in a respective image of the plurality of images (920, 930) displaying a respective calibration mark and located at said each position and displayed on said respective site; and calibrating (216) the optical recording device (150) by at least using sites of the plurality of calibration sites (60-69) as positions in the image plane to a projection on the image plane of the first calibration view.

Description

Technical Field

[0001] This invention relates to a method for calibrating an optical recording device, particularly included in an automated laboratory system (ALS), such as an automated pipetting system. The invention also relates to an automated laboratory system configured to perform the method of the invention. For example, the optical recording device may be a video camera and / or a conventional camera, and is associated with a field of view and an image plane. In particular, the field of view of the device is a portion of a three-dimensional space that can be depicted by the optical recording device. Specifically, the field of view depends on the position and / or orientation of the optical recording device in three-dimensional space. Background Technology

[0002] Methods for calibrating optical recording devices aim to estimate the values ​​of the device's inherent camera parameters. These parameters include, for example, focal length, the two-dimensional coordinates of the image center relative to the image coordinate system, sensor scale, diagonal distortion, and parameters describing lens distortion.

[0003] Calibration methods known in the art estimate the values ​​of camera intrinsic parameters by using a model panel, such as a planar calibration pattern. Typically, this model panel comprises several calibration marks arranged in a known pattern. For example, the calibration marks are the vertices of a two-dimensional chessboard or a two-dimensional mesh drawn on the model panel.

[0004] According to known calibration methods, the values ​​of the camera's inherent parameters are estimated using multiple calibration views. A calibration view comprises a set of calibration positions of the field of view, for example, a set of calibration positions of the field of view. In particular, the calibration positions are locations used to calibrate the camera's field of view. Generally, the positions of the calibration views are arranged in a substantially planar region of the field of view.

[0005] Camera intrinsic parameters are estimated using the coordinates of the calibration view's location and its projection onto the image plane. Typically, the projection of the calibration view's location is estimated from said location, which is depicted in the image associated with the calibration view.

[0006] According to known calibration methods, the image associated with the calibration view depicts the position of the model panel and its orientation relative to the optical recording device, such that the position of the calibration view associated with the image is marked by calibration marks on the model panel. Specifically, each position of the calibration view associated with the image is marked by a corresponding calibration mark on the model panel. Therefore, each calibration view is associated with a single image in which each position of the calibration view is marked by a mark on the model panel, and thus different calibration views are associated with different images.

[0007] Methods known in the art require several calibration views that must meet specific conditions. In particular, the position of at least one calibration view should be able to be mapped to the position of another calibration view through three-dimensional rotation and translation. Therefore, in order for the optical recording device to form an image in which the model panel marks the position of the previous calibration view and an image in which the model panel marks the position of the subsequent calibration view, the model panel must be rotatable and translatable.

[0008] This condition of the model panel increases the complexity of the calibration operation, especially if the model panel's movement is automatic, as the model panel must be equipped with a suitable rotation mechanism. Generally, automatic movement of the model panel increases calibration accuracy and is necessary in some applications. For example, in many ALS systems, the optical recording device is positioned and oriented relative to the stage such that, due to the reduced space between the device and the stage, manual movement of the model panel is not permitted.

[0009] Furthermore, calibration methods typically require calibration views with a relatively large number of calibration positions. Therefore, the model panel is sized to accommodate a large number of calibration marks, which limits the minimum size the model panel can have. In many applications, the relatively large size of the model panel can increase the complexity of the calibration operation, especially if the model panel must be moved within a relatively small space, such as between the optical recording device and the ALS stage. Summary of the Invention

[0010] Therefore, there is a need for a detection method that requires only translational motion of the model panel, and that the detection method can be performed using a relatively small model panel.

[0011] In a first aspect, the present invention relates to a computer-implemented method for calibrating an optical recording apparatus by using at least a model panel and a first calibration view. The optical recording apparatus is associated with a field of view and an image plane, and the model panel includes a set of calibration marks. The first calibration view includes a first set of calibration positions for the field of view.

[0012] The method according to a first aspect of the present invention includes the following steps:

[0013] - Construct at least a first plurality of scenes such that, for each of the first set of calibration positions, a calibration mark in this set of marks is located at the corresponding scene of the first plurality of scenes at each position, and wherein each of the first plurality of scenes is constructed by translating a model panel to a corresponding position relative to the optical recording device; and

[0014] - Acquire at least a plurality of images such that each of the plurality of scenes is displayed in a corresponding image of the plurality of images. In particular, the corresponding image of the plurality of images is associated with each of the plurality of scenes.

[0015] The method of the first aspect of the present invention further includes the following steps:

[0016] - Locate at least a first plurality of calibration sites such that, for each location in the first set of calibration locations, a corresponding site of the first plurality of calibration sites is included in a corresponding image of the first plurality of images, the corresponding image displaying a corresponding calibration mark, and the corresponding calibration mark being located at each location and displayed in the corresponding site. In particular, the location in the first set of calibration locations is associated with the corresponding site of the first plurality of calibration sites.

[0017] Furthermore, the method of the first aspect of the present invention further includes the following steps:

[0018] - The optical recording device is calibrated by using at least one of the first plurality of calibration sites as positions in the image plane projected onto the image plane of the first calibration view.

[0019] In particular, the step of calibrating the optical recording device is performed by using each of the first plurality of calibration sites as a position in an image plane projected onto the corresponding position of a first set of positions associated with each calibration site.

[0020] Optical recording devices may include or consist of conventional cameras and / or video cameras. For example, optical recording devices may include or consist of digital cameras and / or analog cameras. In particular, optical recording devices may include multicolor cameras, monochrome cameras, grayscale cameras, black and white cameras, UV cameras, IR cameras, video cameras, and / or three-dimensional RGBD cameras, or may consist of multicolor cameras, monochrome cameras, grayscale cameras, black and white cameras, UV cameras, IR cameras, video cameras, and / or three-dimensional RGBD cameras.

[0021] For example, the optical recording device may be an intelligent device, such as a recording device for capturing images, a processing element (CPU, GPU, etc.). The optical recording device may include a storage device, comprising volatile main memory (e.g., RAM, DRAM, SRAM, CPU cache memory, etc.), non-volatile main memory (e.g., ROM, PROM, EPROM, etc.), and / or secondary memory. In particular, the optical recording device may be a computing device for performing the method of the present invention.

[0022] The optical recording device can be located at a fixed position in the three-dimensional world and / or at a fixed position relative to an object included in the field of view. For example, if the field of view includes a workbench of an automated laboratory system, the optical recording device can be located at a fixed position relative to the workbench.

[0023] For example, in a pinhole camera model, the image plane is the plane in which the three-dimensional world is projected through the camera's aperture. For instance, if the optical recording device is an analog camera, the image plane is the photosensitive surface of the film onto which light passing through the open shutter illuminates. If the optical recording device is a digital camera, the image plane can be the sensor plane. In particular, the image plane can be a curved surface, for example, considering the geometry of the lens of the optical recording device.

[0024] A position in the image plane can be represented by two coordinates in a reference frame FI that is constrained within the image plane. These coordinates will also be referred to as image plane coordinates in the following text.

[0025] In particular, the field of view of the device is the portion of three-dimensional space that can be depicted by the optical recording device. In particular, the field of view depends on the position and / or orientation of the optical recording device in three-dimensional space.

[0026] A calibration view, specifically a set of locations within a field of view, at least some of which are used to calibrate an optical recording device. The calibration view may be associated with rotational translation and / or homography transformations. These transformations can be used to define the position of the calibration view within the field of view relative to a three-dimensional reference frame F. fix The position of the reference frame is fixed relative to the optical recording device. In particular, rotation, translation, and / or homography transformation can be set by a set of parameters that can be used to perform the calibration of the optical recording device by using at least one of the first plurality of calibration sites.

[0027] The first calibration view may be composed of positions from a first set of positions. For example, the first calibration view is defined by positions from a first set of calibration positions. In particular, the positions from the first set of calibration positions are arranged on a first region of the field of view. For example, the first region may be substantially planar. In particular, the first set of calibration positions includes N positions P. (1),1 ,P (1),2 ,…,P (1),N .

[0028] The positions in the first set of calibration positions can be represented by three coordinates in a first reference frame F1, which is defined in the three-dimensional world, for example, in the field of view. If the positions in the first set of calibration positions are arranged in a first planar region, the first reference frame F1 can be defined such that its (X,Y) plane lies on the first planar region, i.e., the Z coordinate of each position in the first set of calibration positions disappears. In particular, the rotation, translation, and / or homography transformation associated with the first calibration view is to map the coordinates of the first reference frame to the reference frame F1. fix Transformation in coordinates.

[0029] In particular, the scene includes objects located within the field of view of the optical recording device. A scene can be defined by the objects located within the field of view of the optical recording device and by the positions of said objects within the field of view. Two different scenes including the same object may differ due to the position of said object within the field of view. For example, each of the first plurality of scenes differs from the other scenes in the position of the model panel relative to the optical recording device.

[0030] For example, each of the first plurality of scenes includes a model panel that is positioned and oriented in the field of view such that at least one of a set of markers is located at a position in the first set of calibration positions. In particular, in each of the first plurality of scenes, the model panel is arranged such that at least one position in the first set of calibration positions is marked by a marker in that set of markers.

[0031] For example, at least one of the first plurality of scenes includes a model panel that is positioned and oriented in the field of view such that each marker in a subset of the set of calibration markers is located at a corresponding position in the first set of calibration positions. For example, in at least one of the first plurality of scenes, the model panel is arranged such that at least the fifth and sixth positions in the first set of calibration positions are marked by a first marker and a second marker, respectively. Therefore, in particular, the number of scenes in the first plurality of scenes may be less than or equal to the number of positions in the first set of calibration positions.

[0032] According to the present invention, particularly for each of the first set of calibration positions, the calibration mark in that set of marks is located in a corresponding scene of a first plurality of scenes. In this specification, the corresponding scene will also be referred to as the scene associated with each of the positions. One scene of the first plurality of scenes may be associated with multiple positions in the first set of positions.

[0033] According to the present invention, the image can be a vector image or a two-dimensional grid of pixels, such as at least one of a first plurality of images, and the two-dimensional grid of pixels can be, for example, a rectangular grid of pixels. In particular, the position of a pixel in the image can be uniformly determined based on its two-dimensional image coordinates in the image, the coordinates representing the position of the pixel in the two-dimensional grid of pixels.

[0034] An image, such as at least one of a first plurality of images, may be encoded by at least one bitmap. The bitmap encoding the image or a portion thereof may include, for example, a bit array specifying the intensity (e.g., color) of each pixel of the image or a portion thereof. The bitmap may be palette-indexed, such that the entries of the array are indexes to a color table. The entries of the array may store bits encoding the color of the pixels. The bitmap may include a raster data structure, for example, a raster data structure representing a two-dimensional grid of pixels. The bitmap may further include information about the number of bits per pixel, the number of pixels per row of the two-dimensional grid, and / or the number of pixels per column of the grid. An image viewer may use the information encoded in the bitmap to render the image on a screen of a computing device.

[0035] The step of acquiring an image from the first plurality of images can be performed by capturing the image using an optical recording device. This step may further include storing the image from the first plurality of images in, for example, the main memory or secondary memory of a computing device and / or the optical recording device. In particular, the image captured by the optical recording device—that is, at least one image from the first plurality of images—can be an upper limit of video. For example, the image captured by the optical recording device—that is, at least one image from the first plurality of images—can be a frame of video.

[0036] The position of at least one of the first plurality of calibration points can be represented by its image plane coordinates. Specifically, the position of each calibration point in the first plurality of calibration points is represented by its image plane coordinates. For example, the first plurality of calibration points may include N elements M. (1),1 M (1),2 ,…,M (1),N For example, it can consist of N elements M (1),1 M (1),2 ,…,M (1),N Composition. For each j∈{1,2,…,N}, calibration site M (1),j It can be derived from image coordinates Description. In particular, for each j∈{1,2,…,N}, position P (1),j With calibration site M (1),j Related.

[0037] The calibration of an optical recording device can be performed using a calibration algorithm that takes at least one position from a first set of calibration positions and one position from a first plurality of calibration sites as input. Specifically, the calibration algorithm uses each of the first plurality of calibration sites as an observation point, at which the position associated with each site is displayed in the image plane. For example, the calibration of the optical recording device can be performed using Zhang's calibration algorithm and / or Chua's calibration algorithm.

[0038] In one embodiment of the invention, the step of locating the first plurality of calibration sites begins only after the step of constructing the first plurality of scenes is completed and / or after the step of acquiring the first plurality of images is completed. Alternatively, the step of locating sites among the first plurality of calibration sites may be performed in a plurality of sub-steps, and at least one of these sub-steps may be performed before the step of constructing the scenes among the first plurality of scenes is completed and / or before the step of acquiring the images among the first plurality of images is completed. According to the invention, the steps of constructing the first plurality of scenes and / or acquiring the first plurality of images may be performed in a plurality of sub-steps.

[0039] According to the invention, a set includes at least one element, and in particular may consist of a single element. Alternatively, a set may include multiple elements. For example, a set of markers may consist of one marker, or include multiple markers. A first set of calibration positions may include at least one position and / or multiple positions. In particular, a first set of calibration positions may include at least four positions.

[0040] This invention estimates the coordinates of the points in the image plane where the position of the first calibration view is depicted by using a first plurality of images. Each image depicts at least a mark on the model panel at a calibration position within the first set of calibration positions, and each image is constructed by translating the model panel. According to the invention, the calibration positions in the first set of calibration positions do not need to be marked in the same image; therefore, the size of the model panel is not determined by the number of positions in the first set of calibration positions. Thus, optical recording devices can be calibrated using relatively small model panels, especially those small enough to accommodate a single calibration mark.

[0041] Furthermore, the method of the present invention allows for the calibration of the optical recording device without the need to rotate the model panel, as shown below with respect to exemplary embodiments.

[0042] In one embodiment of the method of the present invention, the optical recording device is further calibrated using a second calibration view, which includes a second set of calibration positions in the field of view. Specifically, each position in the second set of calibration positions can be mapped to a corresponding position in the first set of calibration positions using a first rotational translation.

[0043] The second calibration view can be composed of positions from a second set of positions. For example, the second calibration view is defined by positions from a second set of positions. In particular, the positions from the second set of calibration positions are arranged on a second region of the field of view. For example, the second region can be substantially planar. The second set of calibration positions can include at least one position and / or multiple positions. In particular, the second set of calibration positions can include at least four positions. In particular, the second set of calibration positions includes N positions P. (2),1 ,P (2),2 ,…,P (2),N .

[0044] The positions in the second set of calibration positions can be represented by three coordinates in a second reference system F2, which is defined in the three-dimensional world, for example, in the field of view. For example, the second reference system can be the first reference system. If the positions in the second set of calibration positions are arranged in a first planar region, the second reference system F2 can be defined such that its (X,Y) plane lies on the second planar region, i.e., the Z coordinate of each position in the second set of calibration positions disappears. In particular, the rotation, translation, and / or homography transformation associated with the second calibration view maps the coordinates of the second reference system to the reference system F. fix Transformation in coordinates.

[0045] Rotational translation is a proper rigid transformation in the three-dimensional world, which includes rotation and / or translation. In particular, rotational translation is a proper rigid transformation that includes rotation and, optionally, translation. For example, a rotational translation that transforms a first plane into a second plane (the second plane is not parallel to the first plane) should include rotation.

[0046] In a three-dimensional reference frame F, the first rotation and translation can be performed using a 3×3 orthogonal rotation matrix R. F and three-dimensional translation vector Representation. For example, for each j∈{1,2,…,N}, position P (2),j The position P mapped to the first set of calibration positions (1),j In particular, for each j∈{1,2,…,N}, position P (1),j Coordinates in reference frame F Depend on Given, among which It is position P (2),j Coordinates in reference frame F.

[0047] As described above, the method of the present invention allows for the calibration of the optical recording device without rotating the model panel. This is true even when, for example, the model panel is positioned in a third position and oriented in a third orientation in the field of view such that the markings on the model panel indicate all positions in the second set of calibration positions. In particular, the markings on the model panel are located at position P.(2),1 And therefore, the location P is marked. (2),1 In reference frame F, the position P (2),1 From coordinates describe.

[0048] For example, in this case, with position P (1),1 The relevant scenario involves translating the model panel from the third position to a position relative to P using translation. (1),1 The translation is constructed from the associated positions. In reference frame F, the translation is constructed from a three-dimensional translation vector. describe.

[0049] With position P (1),2 The associated scenario involves translating the model panel from position P. (1),1 The associated position is translated to position P. (1),2 The translation is constructed from the associated positions. In reference frame F, the translation is constructed from a three-dimensional translation vector. To describe.

[0050] With position P (1),3 ,P (1),4 ,…,P (1),N The associated scene is constructed in a similar manner by translating the model panel using appropriate translation. Specifically, with respect to the j-th position P... (1),j (j≥2) The associated scenario is to move the model panel from position P by means of translation. (1),j-1 The associated position is translated to position P. (1),j The translation is constructed from the associated positions. In reference frame F, the translation is constructed from a three-dimensional translation vector. describe.

[0051] One embodiment of the present invention further includes the following steps:

[0052] - Construct at least a second plurality of scenes such that, for each position in a third set of calibration positions, the calibration mark in that set of marks is located at the corresponding scene of the second plurality of scenes at each position, and wherein each scene in the second plurality of scenes is constructed by translating a model panel to a corresponding position relative to the optical recording device; and

[0053] - Acquire at least a second or more images such that each scene in the second or more scenes is displayed in the corresponding image of the second or more images.

[0054] In particular, the third set of calibration positions includes at least the positions in the second set of calibration positions that are not included in the first set of calibration positions. For example, the third set of calibration positions is equal to the second set of calibration positions, or the third set of calibration positions consists of positions included in the second set of calibration positions.

[0055] This embodiment may further include the following steps:

[0056] - Locate at least a second plurality of calibration sites such that for each of the third set of calibration locations, the corresponding site of the second plurality of calibration sites is included in the corresponding image of the second plurality of images, the corresponding image displaying the corresponding calibration mark, and the corresponding calibration mark being located at each location and displayed in the corresponding site.

[0057] Furthermore, in this embodiment, the step of calibrating the optical recording device can be performed by using a position from one of the third plurality of calibration points as the position in the image plane projected onto the image plane of the second calibration view.

[0058] In particular, the third plurality of calibration sites includes calibration sites from the second plurality of calibration sites and calibration sites from the first plurality of calibration sites, which satisfy each of the conditions in a set of conditions. For example, if none of the calibration sites in the first plurality of calibration sites satisfy each of the conditions in the set of conditions, then the third plurality of calibration sites may consist of calibration sites from the second plurality of calibration sites.

[0059] For example, this set of conditions includes, for instance, the following conditions: the calibration site is associated with a position in the field of view that is included in the first set of calibration positions and the second set of calibration positions. For example, for each of the third plurality of calibration sites, each site is included in the second plurality of calibration sites, and / or each site is associated with a position included in the first set of calibration positions and the second set of calibration positions.

[0060] An embodiment of the method of the present invention further includes the following steps:

[0061] - The first set of calibration positions is constructed by mapping each position in the second set of calibration positions to the corresponding position in the field of view using a first rotation and translation.

[0062] In another embodiment of the method of the present invention, the positions in the first set of positions are arranged on a first planar region of the field of view, and the positions in the second set of positions are arranged on a second planar region of the field of view. In particular, the first and second planar regions form an angle with each other. For example, the first and second planar regions are substantially perpendicular to each other. More specifically, the first and second planar regions are not parallel to each other.

[0063] In this case, the method allows for the calibration of the optical recording device using two non-parallel views without requiring a rotating model panel.

[0064] In particular, the positions in the first group and the positions in the second group are arranged on the third plane region of the field of view.

[0065] According to one embodiment of the invention, the positions in the first set of calibration positions are arranged relative to each other such that each position in the first set of calibration positions is located at a corresponding node of a first grid diagram. In particular, the first grid diagram is a two-dimensional grid diagram, and more particularly, a rectangular and / or square grid. According to the invention, the grid diagram is in particular a lattice diagram defined by a first lattice constant and a second lattice constant.

[0066] This arrangement of the positions in the first group of calibration locations can reduce the number of scenes in the first plurality of scenes. In particular, several positions in the first group of locations can be marked in the same scene. For example, this can be achieved by using a model panel in which the marks are arranged at the nodes of a mesh diagram that corresponds to, or is substantially equal to, the first mesh diagram. For example, the first lattice constant and the second lattice constant of the latter mesh diagram are equal to the first lattice constant and the second lattice constant of the first mesh diagram, respectively.

[0067] According to one embodiment of the invention, the positions in the second set of calibration positions are arranged relative to each other such that each position in the second set of calibration positions is located at a corresponding node of the second grid diagram. In particular, the second grid diagram is a two-dimensional grid diagram, and more particularly a rectangular and / or square grid. For example, the first lattice constant and the second lattice constant of the second grid diagram are equal to the first lattice constant and the second lattice constant of the first grid diagram, respectively. For example, the second grid diagram can be mapped to the first grid diagram using a first rotational translation. In one embodiment, the first lattice constant and / or the second lattice constant of the second grid diagram may be different from the first lattice constant and / or the second lattice constant of the first grid diagram.

[0068] In one embodiment of the invention, the first set of calibration positions includes at least a first position and a second position, the first position and the second position being aligned with each other on a first line at a first distance, and the set of calibration marks includes at least a first mark and a second mark, the first mark and the second mark being aligned with each other on a second line at a second distance.

[0069] In particular, the first distance and the second distance are essentially equal to each other, and the first line and the second line are essentially parallel to each other.

[0070] In this embodiment, the step of constructing the first plurality of scenes includes constructing the first scene by means of the following steps: translating the model panel to a first position relative to the optical recording device, such that in the first position, the first mark and the second mark are located at the first position and the second position, respectively.

[0071] In particular, the first set of calibration positions includes at least a first plurality of calibration positions aligned on a first line, and the calibration markings include at least a first plurality of markings aligned on a second line. For example, in the first position, each of the first plurality of markings is located at a corresponding position of the first plurality of calibration positions.

[0072] In this embodiment, the first and second positions are marked in the same scene and therefore depicted in the same image. Thus, the number of scenes in the first plurality of scenes, and consequently the number of images in the first plurality of images, is reduced, and the calibration process is simplified.

[0073] In one embodiment of the invention, the first set of calibration positions includes at least a third position and a fourth position. The third and fourth positions are aligned on a third line, which is substantially parallel to the first line. According to this embodiment, the first and third positions are aligned on a fourth line, and the second and fourth positions are aligned on a fifth line, which is substantially parallel to each other.

[0074] The steps of constructing the first plurality of scenes may include constructing the second scene by translating the model panel from a first position to a second position relative to the optical recording device, such that in the second position, the first mark and the second mark are located in a third position and a fourth position, respectively. In particular, the first set of calibration positions may include a second plurality of calibration positions aligned on a third line, and the calibration marks include a second plurality of marks aligned on a second line. Specifically, in the second position, each of the second plurality of marks is located at a corresponding position in the second plurality of calibration positions. For example, the second plurality of marks may include or be incorporated into the first plurality of marks. The second plurality of marks may consist of the first plurality of marks.

[0075] In this embodiment, the number of scenes in the first plurality of scenes, and therefore the number of images in the first plurality of images, is further reduced. In particular, in this case, the third position and the fourth position are marked in the same scene and therefore depicted in the same image.

[0076] In one embodiment of the present invention, the translation vector It is essentially parallel to the fifth and fourth lines. Therefore, the movement of the model panel between the first and second positions is minimized. This simplifies the calibration process and reduces the energy consumption of performing it. For example, in reference frame F, the translation vector... Depend on Provided.

[0077] In one embodiment of the invention, the set of calibration marks consists of a third mark, and optionally, the third mark is a point or a polygon, especially a square.

[0078] In this case, the size of the model panel can be reduced because the panel includes a single marker.

[0079] According to one embodiment of the invention, calibration marks in a set of calibration marks are arranged relative to each other on a model panel such that each calibration mark in the set is located at a corresponding node of a third mesh diagram. In particular, the third mesh diagram is a two-dimensional mesh diagram, and more particularly a rectangular and / or square mesh. For example, the first and second lattice constants of the third mesh diagram are equal to the first and second lattice constants of the first mesh diagram, respectively.

[0080] According to the present invention, the model panel may include a plurality of squares forming a square grid or checkerboard pattern. If present, the square grid or checkerboard pattern may include a set of vertices of the plurality of squares, wherein each calibration mark in a set of calibration marks is a corresponding vertex of the set of vertices or is located at a corresponding vertex of the set of vertices.

[0081] In one embodiment of the invention, the step of locating the first plurality of calibration sites includes locating at least one site among the first plurality of calibration sites using an image recognition algorithm. Specifically, each site among the first plurality of calibration sites is located using an image recognition algorithm.

[0082] An algorithm is, in particular, a set, such as a sequence, of instructions for processing input information to obtain output information. The instructions of the algorithm can be implemented in a computer and executed, for example, by a processor of a data processing system according to the invention. In particular, the input information of the algorithm is encoded in input data that can be accessed by the processor executing the algorithm. In particular, the processor processes the input data according to the instructions of the algorithm to produce output information, which is typically encoded in output data. According to the invention, in particular, when the algorithm includes instructions, the algorithm processes data, which, when executed by the processor, cause the processor to process the data.

[0083] For example, an image recognition algorithm processes input data, which includes the position and intensity value of each pixel in an input image or a portion thereof. In particular, the image recognition algorithm includes instructions that, when executed by a processor, cause the processor to process the input data to attempt to detect one or more markers from a set of markers in the input image.

[0084] An image recognition algorithm can be an algorithm that attempts to detect one or more markers from a set of markers in an input video captured by an optical recording device. For example, an image recognition algorithm includes instructions that, when executed by a processor, cause the processor to process frames of the input video to attempt to detect one or more markers from a set of markers in the frames.

[0085] The output information may include information indicating whether one or more of the markers in this set are displayed in the input image. If so, the output information may further include information indicating the location at which the detected marker is displayed in the image.

[0086] According to one embodiment of the method, the image recognition algorithm includes at least a machine learning algorithm. Specifically, the machine learning algorithm includes instructions for processing input information to obtain output information, and at least some of these instructions are set using a set of training data and a training algorithm. The machine learning algorithm may include artificial neural networks (ANNs), decision trees, random forests, support vector machines (SVMs), etc. For example, the first machine learning algorithm may be a convolutional neural network and / or a deep neural network.

[0087] In one embodiment of the invention, the step of calibrating the optical recording device is performed using a set of parameters, including parameters that parameterize a second rotational translation and / or homography transformation. Specifically, the second rotational translation and / or homography transformation is associated with a first calibration view. For example, the second rotational translation and / or homography transformation can be used to define a position in a first set of calibration positions within a field of view relative to a reference frame in the three-dimensional world. In particular, the reference frame is fixed relative to the optical recording device.

[0088] This set of parameters may include parameters suitable for uniquely determining the second rotation and translation, for example, it may consist of parameters suitable for uniquely determining the second rotation and translation. In particular, this set of parameters consists of A elements, which may be collected in an A-dimensional list. The list is given as follows:

[0089]

[0090] For example, the second rotation and translation can be represented by a 3×3 orthogonal rotation matrix and a translation vector.

[0091]

[0092] In this case, the set of parameters can include translation vectors. The three entries and the rotation matrix R (1)The three parameters are used to parameterize the entries. For example, these three parameters could be entries for the Rodriguez vector, or entries for the rotation matrix R. (1) The three Euler angles are parameterized.

[0093] In reference frame F, homography transformation can be performed using a 3×3 homography matrix H. (1),3 and / or a 3×4 homography matrix H (1),4 In this case, the set of parameters can include H. (1),3 The nine entries and / or H (1),4 The twelve entries of the homography matrix. These homography matrix entries can be parameterized with parameters that parameterize the second rotation and translation, for example, with matrix R. (1) sum vector The entries are parameterized. For example,

[0094]

[0095] and / or

[0096]

[0097] Where λ is the overall scaling factor, f is the focal length of the optical recording device, (u c v c ) T It is the position of the image center, s θ It is radial distortion. x and s y These are the second scales in the x and y directions, respectively. These camera calibration parameters can be collected in a six-dimensional list. In the middle, a six-dimensional list Given from the following:

[0098]

[0099] In one embodiment of the invention, the step of calibrating the optical recording device is performed using a first set of parameterized points, each of which is a parameterization of the position of the projection onto the image plane to the corresponding position of the first set of calibration positions. Each parameterized point in the first set of parameterized points may depend on at least one parameter in the set.

[0100] In particular, a set of parameterized sites can include N elements Q. (1),1 Q (1),2 ,…,Q (1),N For each j∈{1,2,…,N}, the parameterized site Q (1),j It can be derived from image coordinates q (1),j =(u (1),j v(1),j )T description.

[0101] For example, for each j∈{1,2,…,N}, the image coordinates q (1),j It can be a function of calibration parameters and coordinates (X′ Y′ Z′).

[0102]

[0103] The coefficient D(X′,Y′,Z′)=1 may be a function of the scalar r, which is given by the following:

[0104]

[0105] For example, the coefficients D(X′,Y′,Z′) can be given by the following:

[0106] D(X′,Y′,Z′)=1,(7a)or

[0107]

[0108] In particular, K can be equal to 2, 3, or 4. The coefficient k c These are the distortion coefficients of the optical recording device. These coefficients can be collected in a K-dimensional list. In the K-dimensional list As given below:

[0109]

[0110] For example, if lens distortion is nonexistent or negligible, then D(X′,Y′,Z′) can be given by formula (7a).

[0111] In particular, the coordinates (X′ Y′ Z′) can be expressed in terms of position P. (1),j coordinates express:

[0112]

[0113] As discussed above, the rotation matrix R (1) Translation vector It can be a set of parameters, such as an A-dimensional list. The parameters included are used for parameterization. In particular, the rotation matrix R (1) Translation vector This describes mapping a reference frame F to a camera reference frame F. C Rotation and translation within the frame. Specifically, the reference frame F. CIt is defined in a three-dimensional world, and such that: (i) its origin is located at the optical center of the optical recording device; (ii) its Z-axis is aligned with the optical axis of the optical recording device; and (iii) its Z-axis is perpendicular to the image plane. In particular, in reference frame F... C In the image plane, the Z-axis intersects at point (0 0 f). T They intersect.

[0114] For example, for each j∈{1,2,…,N}, the image coordinates q (1),j It can be a function of the aforementioned homography matrix:

[0115]

[0116] In particular, the latter formula holds in the reference frame, where Z (1),j = 0. For example, if the reference frame F is chosen such that for every j∈{1,2,…,N}, Z (1),F,j =0, matrix H (1),3 The value of each entry can be obtained by using coordinates. Image coordinates m (1),1 m (1),2 ... m (1),N And estimation can be achieved through direct linear transformations. In this case, matrix H (1),3 The entries can be parameterized according to formula (3a). Therefore, in this case, matrix H (1),3 The numerical values ​​of the entries can be used to estimate the list. The numerical values ​​of each calibration parameter and matrix R included in the data. (1) The numerical values ​​and vectors of each entry in the first two columns. The numerical value of each entry.

[0117] For example, for each j∈{1,2,…,N}, the image coordinates q (1),j It can be a function of calibration parameters and coordinates (X′ Y′ Z′):

[0118]

[0119] Among them, f x and f y Focal length is expressed in pixels. The coordinates (X″ Y″ Z″) can be represented as follows:

[0120]

[0121] The coefficients are represented by the coordinates (X′ Y′ Z′) defined in formula (9) and the quantity r defined in formula (6). arrive It is the radial distortion coefficient, the coefficient and It is the tangential distortion coefficient.

[0122] The use of parameterized sites allows for the determination of a position within a first set of calibration positions using the (known) coordinates of the markings in the calibration panel and the position and orientation of a first region of the field of view. The position and orientation of the first region are described by a second rotational translation and / or homography transformation, which is parameterized by parameters from a set of parameters. These parameters can be estimated during the calibration steps, for example, using Zhang's calibration algorithm and / or Chua's calibration algorithm, without needing prior knowledge or measurement before the calibration steps, thereby simplifying the calibration process.

[0123] In particular, the steps of calibrating an optical recording device include minimizing a parameter function relative to a set of parameters. This parameter function depends on the parameters in this set of parameters.

[0124] In particular, the parametric function depends on the sites among multiple calibration sites and the parameterized sites among a set of parameterized sites. More specifically, the parametric function depends on the parameters in that set of parameters through its correlation with the parameterized sites among the set of parameterized sites.

[0125] In particular, at least for the camera's calibration parameters and / or parameters within this set of parameters, the parametric function is minimized. In particular, at least for the list... and list The parameters included, and if they exist, for a list. The parameters included are used to minimize the parametric function. Minimization can be achieved using iterative methods, such as the Levenberg-Marquardt method.

[0126] For example, the parameter functions include the function G given below. (1) :

[0127]

[0128] Among them || v || [2] Representing a two-dimensional vector v The norm function, q (1),j It can be calculated using formulas (5) to (9) or formulas (6), (9), (11), and (12). Function G (1) It can depend on the list The calibration parameters included, and if present, depend on the list. The calibration parameters in the coordinates q (1),1 ,q (1),2 ,…, q (1),N These parameters may depend on the equations (5) through (9). These coordinates and therefore the function G (1) It can also depend on the parameters in a set of parameters.

[0129] If the reference frame F is chosen such that for every j∈{1,2,…,N}, Z (1),F,j =0, the parameter function can include the function L given below. (1) :

[0130]

[0131] in Representing a three-dimensional vector The norm function. In this case, the parameter function can be at least for matrix H. (1),3 The entries are minimized. This minimization allows for the estimation of the numerical value of each of these entries. As explained above, these numerical values ​​allow for the estimation of the list. The values ​​of each calibration parameter included, and R (1) The values ​​of each entry in the first two columns, and The numerical value of each entry.

[0132] Minimizing the calibration function allows for an increase in the number of calibration locations, thereby reducing the influence of statistical errors that affect the calibration error, such as those related to the uncertainty of the position of the model panel in the field of view and / or the position of the site in the images of the first plurality of images.

[0133] According to one embodiment of the method of the invention, the optical recording device is further calibrated using a third calibration view, which includes a fourth set of calibration positions of the field of view. In particular, each position in the fourth set of calibration positions can be mapped to a corresponding position in the second set of calibration positions using a third rotational translation.

[0134] In particular, the method of the present invention may further include the step of mapping each of the fourth set of calibration positions to a corresponding position in the field of view by using a third rotational translation, thereby constructing a second set of calibration positions.

[0135] In one embodiment of the invention, the optical recording device is further calibrated using multiple calibration views. Specifically, each of the multiple calibration views includes a corresponding set of calibration positions for the field of view. In particular, each position in the corresponding set of calibration positions can be mapped to a corresponding position in the first set of calibration positions using a corresponding rotation and translation. The multiple calibration views may include at least four calibration views, and in particular a second calibration view and / or a third calibration view.

[0136] This invention relates to a data processing system comprising: an optical recording device associated with a field of view and an image plane; a translation device for translating a model panel within the field of view; and a processing device configured to perform the method according to the invention. In particular, the translation device is configured to translate the model plane at least within the field of view. The data processing system may further include a model panel.

[0137] The present invention also relates to an automated laboratory system incorporating the data processing system of the present invention. In particular, the automated laboratory system of the present invention includes an optical recording device associated with a field of view and an image plane, and a translation device for translating a model panel within the field of view. The automated laboratory system of the present invention further includes a processing device configured to perform the method according to the present invention and a worktable for positioning laboratory items. The automated laboratory system of the present invention may further include a model panel.

[0138] Laboratory articles may include or consist of containers for use in clinical or laboratory settings. These containers may be made of glass, plastic, metal, etc. For example, laboratory articles may include or consist of petri dishes, which may include a top, a lid, and / or a bottom. In particular, laboratory articles include, or consist of, sample vials and / or test tubes. Laboratory articles may be for single use, multiple use, and / or disposable. For example, laboratory articles may include plates, pipette tips, tubes, reservoirs, tip holders, height adapters, reservoir racks, and / or tube racks, or consist of, or consist of, plates, pipette tips, tubes, reservoirs, tip holders, height adapters, reservoir racks, and / or tube racks.

[0139] The present invention also relates to a computer program product comprising instructions that, when executed by a data processing system according to the present invention, cause the system to perform a method according to the present invention.

[0140] The present invention relates to a computer-readable storage medium comprising instructions which, when executed by a system according to the invention, cause the system to perform a method according to the invention.

[0141] According to the present invention, a computing device may include processing elements (such as CPUs and GPUs) and storage devices. The storage devices may include at least volatile main memory (e.g., RAM, DRAM, SRAM, and CPU cache memory), non-volatile main memory (e.g., ROM, PROM, and EPROM), and / or secondary memory. In particular, the volatile main memory temporarily stores program files and related data for execution by the processing elements, and the non-volatile main memory may contain bootstrap code for the operating system of the computing device. According to the present invention, the computing device may be a computer system including input and / or output devices, and more particularly, may be a smartphone, computer, or tablet computer. In particular, the first, second, third, and / or fourth nodes of a computer network may be computing devices or a cluster thereof. Attached Figure Description

[0142] Exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings. These drawings and corresponding detailed descriptions are provided only to enable a better understanding of the invention and do not constitute any limitation on the scope of the invention as defined in the claims. In particular:

[0143] Figure 1a This is a schematic diagram of a data processing system according to a first embodiment of the Automated Laboratory System (ALS) of the present invention;

[0144] Figures 1b to 1d This is a schematic diagram of a first embodiment of the ALS according to the present invention;

[0145] Figure 2 This is a flowchart of the operation of the first embodiment of the method according to the present invention;

[0146] Figures 3a to 3c This is a schematic diagram of the first embodiment of ALS;

[0147] Figures 4a to 4c This is a schematic diagram of the first embodiment of ALS;

[0148] Figures 5a to 5c This is a schematic diagram of the first embodiment of ALS;

[0149] Figure 6a and Figure 6b This is a schematic diagram of the first embodiment of ALS;

[0150] Figure 7a and Figure 7b This is a schematic diagram of the first embodiment of ALS; and

[0151] Figure 8a and Figure 8bThis is a schematic diagram of an image captured by performing the method of the present invention in a first embodiment. Detailed Implementation

[0152] Figures 1b to 1d This is a schematic diagram of a first embodiment of the ALS100 according to the present invention. Figure 1c and Figure 1d They are along Figure 1b Sectional view along line 1-1 and sectional view along line 2-2.

[0153] The ALS100 includes an optical recording device in the form of a digital camera 150. The camera 150 is specifically configured to capture images of a first plurality of images and may include or consist of a conventional camera and / or a video camera. The ALS100 also includes a worktable 110 for positioning one or more laboratory items. Figure 1c As best shown, workbench 110 includes waste container 111. ALS 100 includes model panel 140, which in this embodiment has a substantially parallelogram shape. Figure 1c As shown, the model panel 140 includes a set of calibration marks 40-49, which are arranged on the surface 141 of the model panel 140 facing the optical recording device 150. See [reference needed] Figure 1b Each of the markers in this set 40-49 is placed at the corresponding node of the rectangular grid diagram.

[0154] Model panel 140 is operatively connected to translation device 120, which is adapted to translate model panel 140 at least within the field of view of optical recording device 150. For example, translation device 120 may include a first track, a second track, and a first motor (not shown) operatively connected to the second track. The first and second tracks extend substantially parallel to a first direction 310 and a second direction 320, respectively. In particular, the second track is movably connected to the first track such that the second track can be translated by the first motor along the first track, for example, parallel to the first direction 310.

[0155] The translation device 120 includes an extendable arm 121, such as a telescopic arm. Furthermore, the translation device 120 may also include a second motor and a third motor (not shown), operatively connected to the arm 121. The arm 121 extends parallel to a third direction 330, is connected to a model panel 140, and moves together with said panel 140. The arm 121 is movably connected to a second track, allowing the arm 121 to be translated along the second track by the second motor. The arm 121 can be extended along an extension direction 240 by a third motor. Figure 1bAs shown, the extension direction is substantially parallel to the third direction 330. The first motor, the second motor, and / or the third motor can be a servo motor and / or a stepper motor.

[0156] The ALS100 includes a Data Processing System (DPS) 160, which... Figure 1a The diagram is schematically shown. DPS160 may include a computing device or a cluster thereof. For example, DPS160 may be a smartphone, desktop computer, tablet computer, or laptop computer. DPS160 includes a processing element 170 and a storage device 180, which communicate data with each other. Processing element 170 may include a CPU and / or a GPU, or be composed of a CPU and / or a GPU, and includes several modules 171-174 configured to perform the steps of the method of the present invention.

[0157] Storage device 180 may include volatile main memory 181 (e.g., RAM, DRAM, SRAM, or CPU cache memory) and / or non-volatile main memory 182 (e.g., ROM, PROM, or EPROM). In particular, the volatile main memory may consist of RAM. For example, volatile main memory 181 temporarily stores program files and related data for execution by the processing element, and non-volatile main memory 182 may contain boot code for the operating system of DPS 160.

[0158] The storage device 180 may further include a secondary memory 183, which may store an operating system and / or instructions for an algorithm to perform the methods of the present invention. Furthermore, the secondary memory 183 may store a computer program product including instructions that, when executed by the DPS 160, cause the DPS 160 to perform the methods according to the present invention.

[0159] Secondary memory 183, main memory 181, 182, and processing element 170 do not need to be physically housed in the same housing, but can be spatially separated from each other. In particular, secondary memory 183, main memory 181, 182, and processing element 170 can be spatially separated from each other and can exchange data with each other via wired and / or wireless media (not shown).

[0160] DPS160 may include an input / output (I / O) interface 161 that allows DPS160 to communicate with input / output devices such as a display, keyboard, touchscreen, printer, mouse, or camera. DPS160 may further include a network interface controller (NIC) 162 configured to connect DPS160 to a suitable network (not shown). According to the invention, a suitable network may be, for example, an intranet, the Internet, or a cellular network.

[0161] Camera 150 can be connected to processing element 110 via I / O interface 161. For example, camera 150 can be wirelessly connected to the I / O interface via NIC 162. Camera 150 may be an intelligent device with its own memory for storing associated instructions and data for use with I / O interface 161 or peripheral devices.

[0162] Processing element 170 includes a plurality of modules 171 to 174 configured to perform the method of the present invention. Construction module 171 is configured to construct a scene, for example, a scene among a first plurality of scenes. In particular, construction module 171 is configured to control the translational movement of model panel 140, thereby constructing a scene, for example, a scene among the first plurality of scenes. More particularly, construction module 171 is configured to operate a first motor, a second motor, and / or a third motor to translate and / or extend arm 121, thereby translating model panel 140. Acquisition module 172 is configured to operate camera 150 to acquire images, for example, images among a first plurality of images.

[0163] The positioning module 173 is configured to locate at least a first plurality of calibration points. The calibration module 174 is configured to calibrate the optical recording device by using at least one of the first plurality of calibration points as a position in the image plane projected onto the image plane of the first calibration view.

[0164] The ALS100 can be an automated pipetting system. In this case, the ALS100 includes a pipette tip (not shown) for transferring liquids. The pipette tip can be moved relative to the stage by a servo motor and / or a stepper motor. In particular, the DPS160 may include a module configured to operate a motor that moves the pipette tip.

[0165] Figure 2 This is a flowchart 200 illustrating the operation of a first embodiment of the method according to the present invention. In particular, this embodiment of the method can be derived from the methods described above and... Figures 1a to 1d The first embodiment of ALS100, schematically shown in the figure, is used for implementation.

[0166] In step 201, ALS100 constructs a first scene, which in Figures 3a to 3c It is shown schematically in the middle. Figure 3b and Figure 3c They are along Figure 3a The cross-sectional view along line 1-1 and the cross-sectional view along line 2-2. In particular, in step 201, DPS160 controls translation device 120 to translate model panel 140 to a first position relative to camera 150, thereby constructing a first scene.

[0167] like Figure 3b As best shown, in the first scenario, each calibration mark in a set of marks 40-49 is positioned at the corresponding location in the fourth set of calibration positions 10-19. Specifically, in Figures 3a to 3c In the figure, the calibration mark identified by reference numeral n is located at the position identified by reference numeral n-30. Figure 6a and Figure 6b In the diagram, the fourth group of calibration positions 10-19 are marked with shaded dots. Specifically, Figure 6b It is a sectional view along line 1-1 and along Figure 6a Sectional view of line 2-2.

[0168] The positions of the fourth group of calibration positions 10-19 are included in and specifically define the third calibration view. In particular, in this embodiment, the third calibration view includes ten positions P. (3),1 ,P (3),2 ,…,P (3),10 Where, for each j∈{1,2,…,10}, position P (3),j exist Figure 3b It is identified by the attached figure label (j+9).

[0169] In step 202, DPS160 operates camera 150 to acquire a first image 910, which is 190 in Figure 8a The middle part is symbolically represented.

[0170] In step 203, ALS100 locates the calibration sites among the fourth plurality of calibration sites 50-59, such that for each location in the first set of calibration positions 10-19, the corresponding site of the fourth plurality of calibration sites 50-59 is included in the first image 910. Specifically, for each n∈{10,11,…,19}, Figure 3b The position marked by the figure's reference numeral n in the figure is the same as that in the figure. Figure 8a The corresponding sites identified by the attached map marker n+40 are associated with this.

[0171] For each of the first set of calibration positions 10-19, the first image 910 displays a corresponding calibration mark located at each position and shown in the corresponding site associated with each position. Specifically, for each n∈{10,11,…,19}, Figure 3b The position indicated by the reference numeral n in the attached figure is associated with the corresponding calibration mark. Figure 3b In this context, the calibration mark is identified as reference numeral n+30.

[0172] The calibration sites in the fourth plurality of calibration sites 50-59 can be located in the first image 910 by using image recognition algorithms, such as machine learning image recognition algorithms.

[0173] In this embodiment, the third plurality of calibration sites 50-59 consists of ten calibration sites M (3),1 M (3),2 ,…,M (3),10 Composition, where for each j∈{1,2,…,10}, the calibration site M (3),j From image coordinates m (3),j Description, and Figure 8a It is identified by the attached figure label (j+49).

[0174] In step 204, ALS100 constructs the first scene in the first plurality of scenes, the first scene in Figures 4a to 4c The middle part is shown schematically. In particular, Figure 4b and Figure 4c These are the sectional views along line 1-1 and along... Figure 4a A sectional view of line 2-2. In step 204, DPS160 controls translation device 120 to translate model panel 140 from a first position relative to camera 150 to a second position relative to camera 150, thereby constructing a first scene in a first plurality of scenes.

[0175] like Figure 4a and Figure 4b As best shown in the first scenario of the first plurality of scenarios, each calibration mark of the first subgroup 40-44 of a set of marks 40-49 is arranged at the corresponding position of the first subgroup 20-24 of the first set of calibration positions 20-29. In particular, in Figures 4a to 4c In the figure, the calibration mark identified by reference numeral n is located at the position identified by reference numeral n-20. Figure 6a and Figure 6b In the diagram, the positions of the first subgroup 20-24 of the first set of calibration positions 20-29 are marked by shaded squares. As best shown in the following diagram, in order to construct the first scene in the first set of multiple scenes, the model panel 140 is translated by translation vector 70, such that when the model panel 140 is translated from the first position to the second position, the calibration mark 40 moves from position 10 to position 20.

[0176] In step 205, DPS160 operates camera 150 to acquire the first image 920 among the first plurality of images 920, 930, referring to... Figure 8a .

[0177] In step 206, ALS100 locates the calibration sites of the first subgroup 60-64 of the first plurality of calibration sites 60-69, such that for each position of the first subgroup 20-24 of the first group of calibration positions 20-29, the corresponding site of the first subgroup 60-64 of the first plurality of calibration sites 60-69 is included in image 920. Specifically, for each n∈{20,21,…,24}, Figure 4b The position identified by the attached figure mark n and in Figure 8a The corresponding sites identified by the attached map marker n+40 are associated with this.

[0178] For each position in the first subgroup 20-24 of the first set of calibration positions 20-29, image 920 shows the corresponding calibration mark, which is located at each position and displayed in the corresponding site associated with each position. Specifically, for each n∈{20,21,…,24}, Figure 4b The position identified by the attached figure mark n and in Figure 4b The corresponding calibration mark, identified by the reference numeral n+20 in the attached figure, is associated with this.

[0179] By using an image recognition algorithm, the calibration sites of the first subgroup 60-64 of the first plurality of calibration sites 60-69 can be located in the first image 920 of the first plurality of images 920, 930.

[0180] In step 207, ALS100 constructs the second scene of the first plurality of scenes, which is schematically shown in figures 5a to 5c. In particular, Figure 5b and Figure 5c They are along Figure 5a Sectional view of line 1-1 and along Figure 5a The sectional view of line 2-2. In particular, in step 207, DPS160 controls translation device 120 to translate model panel 140 from a second position relative to camera 150 to a third position relative to camera 150, thereby constructing a second scene in a first plurality of scenes.

[0181] like Figure 5a and Figure 5b As best shown in the diagram, in the second scenario of the first plurality of scenarios, each calibration mark of the first subgroup 40-44 of a set of marks 40-49 is arranged at the corresponding position of the second subgroup 25-29 of the first set of calibration positions 20-29. In particular, in Figures 5a to 5c In the figure, the calibration mark identified by reference numeral n is located at the position indicated by reference numeral n-15.

[0182] exist Figure 6a and Figure 6bIn the diagram, the positions of the second subgroup 25-29 of the first group of calibration positions 20-29 are marked by shaded squares. As best shown in the following diagram, to construct the second scene in the first plurality of scenes, the model panel 140 is translated by translation vector 71, such that when the model panel 140 is translated from the second position to the third position, the calibration mark 40 moves from position 20 to position 25. The length of vector 71 is equal to the distance between mark 40 and mark 45, such that the distance between position 20 and position 25 is equal to the distance between position 10 and position 15.

[0183] like Figure 6a and Figure 6b As shown in the optimal configuration, the positions in the first set of calibration positions 20-29 are arranged on the first planar region 720, and the positions in the fourth set of positions 10-19 are arranged on the fourth planar region 710. The first planar region 720 and the fourth planar region 710 form an angle 760° with each other. The fourth planar region 710 can be mapped onto the first planar region 720 by rotation and translation.

[0184] The positions in the first set of calibration positions 20-29 are included in, and specifically define, the first calibration view. In particular, in this embodiment, the first calibration view includes ten positions P. (1),1 ,P (1),2 ,…,P (1),10 Where, for each j∈{1,2,…,10}, Figure 6a and Figure 7a Position P is indicated by the reference numeral (j+19) in the attached diagram. (1),j .

[0185] In step 208, DPS160 operates camera 150 to acquire the second image 930 from the first plurality of images 920, 930, referring to... Figure 8a .

[0186] In step 209, ALS100 locates the calibration sites of the second subgroup 65-69 of the first plurality of calibration sites 60-69, such that for each location of the second subgroup 25-29 of the first set of calibration locations 20-29, the corresponding site of the second subgroup 65-69 of the first plurality of calibration sites 60-69 is included in image 930. Specifically, for each n∈{25,26,…,29}, Figure 5b The position identified by the attached figure mark n and in Figure 8a The corresponding sites identified by the attached figure marker n+40 are associated with this.

[0187] For each position in the second subgroup 25-29 of the first set of calibration positions 20-29, image 930 shows the corresponding calibration mark, which is located at each position and displayed in the corresponding site. Specifically, for each n∈{25,26,…,29}, Figure 5b The position identified by the attached figure mark n and in Figure 5b The corresponding calibration mark, identified by the reference numeral n+15 in the attached figure, is associated with this.

[0188] The calibration sites of the second subgroup 65-69 of the first plurality of calibration sites 60-69 can be located in the second image 930 of the first plurality of images 920, 930 by using an image recognition algorithm.

[0189] In this embodiment, the first plurality of calibration sites 60-69 consists of ten calibration sites M (1),1 M (1),2 ,…,M (1),10 Composition. For each j∈{1,2,…,10}, calibration site M (1),j From image coordinates m (1),j Description, and Figure 8a It is identified by the attached figure mark (j+59).

[0190] According to a first embodiment of the method of the present invention, the localization of the calibration sites of the second plurality of calibration sites 90-99 is performed by repeating steps 212 to 214 five times. In each of these repetitions, a scene in the second plurality of scenes is constructed (step 212), a corresponding image in the second plurality of images 940-980 is acquired (step 213), and the calibration sites of the corresponding subgroups of the second plurality of calibration sites 90-99 are located in the corresponding image (step 214). The general repetition of steps 212 to 214 is described by a counter s, which is initialized to a value of zero and less than six in step 210.

[0191] In step 212, which is repeated for the sth time, ALS100 constructs the sth scene in the second plurality of scenes. In particular, in step 212, which is repeated for the sth time, DPS160 controls the translation device 120 to translate the model panel 140 from the (s+2)th position relative to the camera 150 to the (s+3)th position relative to the camera 150, thereby constructing the sth scene in the second plurality of scenes.

[0192] In the first repetition, s = 1, and the (s+2)th position relative to camera 150 is the third position relative to the camera, i.e., the position of model panel 140 in the second scene within the first plurality of scenes. If s > 1, then the (s+2)th position of model panel 140 relative to camera 150 is specifically the position of model panel 140 in the (s-1)th scene within the second plurality of scenes, which is constructed in step 212 of the (s-1)th repetition.

[0193] exist Figure 7a and 7b In the second group of calibration positions 30-39, the positions are marked by shaded triangles. Specifically, Figure 7b It is along Figure 7a The sectional view of line 2-2. In the s-th scene of the second plurality of scenes, calibration mark 40 is arranged on... Figure 7a and 7b The location is identified by the reference numeral (s+29) in the attached diagram. In this scenario, calibration mark 45 is arranged at... Figure 7a and 7b The location identified by the reference numeral (s+34) in the attached diagram. For example... Figure 7a and 7b As shown in the best example, in order to construct the s-th scene in the second plurality of scenes, the model panel 140 is translated by the translation vector identified by the attached label (s+79).

[0194] For example, in the first repetition (s=1), calibration marker 40 and calibration marker 45 are positioned at positions 30 and 35, respectively. For example, to construct the first scene in a second set of multiple scenes (step 212 of the first repetition), model panel 140 is translated by translation vector 80, such that when model panel 140 is translated from the third position to the fourth position, calibration marker 40 moves from position 25 to position 30 (see...). Figure 7a , 7b Translation vectors 81-84 have the same length, which is in particular equal to the distance between markers 40 and 41.

[0195] like Figure 7a As shown in the optimal diagram, the positions in the second set of calibration positions 30-39 are arranged on the second planar region 730. The first planar region 720 and the second planar region 730 form an angle with each other, and the second planar region 730 can be mapped onto the first planar region 720 by rotation and translation.

[0196] The positions in the second set of calibration positions 30-39 are included in the second calibration view, and in particular, define the second calibration view. Specifically, in this embodiment, the second calibration view includes ten positions P. (2),1 ,P (2),2 ,…,P(2),10 For each j∈{1,2,…,10}, position P (2),j exist Figure 7b It is identified by the attached figure label (j+29).

[0197] In the s-th repetition of step 213, DPS160 operates camera 150 to acquire the s-th image among the second plurality of images 941-945. Figure 8a The first image 941 in a second set of images is schematically shown. Figure 8b In the second set of images, the s-th image (s>1) is identified by the reference numeral (s+940).

[0198] In step 214, repeated for the sth time, ALS100 locates the calibration sites of the sth subgroup of the second plurality of calibration sites 90-99 in the sth image of the second plurality of images 941-945. These calibration sites can be located in the sth image of the second plurality of images 941-945 using an image recognition algorithm.

[0199] The second set of multiple calibration sites 90-99, the s-th subgroup, is composed of... Figure 8a or Figure 8b The calibration sites are marked by the appendix label (s+89) and in Figure 8a or Figure 8b The calibration sites are identified by the reference numeral (s+94). For example, the first subgroup of the second plurality of calibration sites 90-99 consists of calibration sites 90 and 95 of the first image 941 in the second plurality of images 941-945, as shown in the reference. Figure 8a .

[0200] In particular, Figure 7b The position identified by the attached figure mark (s+29) and Figure 8a or Figure 8b The calibration sites identified by the attached reference numeral (s+89) are associated with this. Figure 7b The position identified by the attached figure mark (s+34) and Figure 8a Or, they are associated with the calibration sites identified by reference numeral (s+94) in Figure 8b. For example, positions 30 and 35 are associated with calibration sites 90 and 95, respectively (see Figure 8b). Figure 7b and 8a ).

[0201] In this embodiment, the second plurality of calibration sites 90-99 consists of ten calibration sites M (2),1 M (2),2 ,…,M (2),10 Composition, where for each j∈{1,2,…,10}, the calibration site M (2),j From image coordinates m (2),jDescription, and Figure 8a or Figure 8b It is marked with (j+89) in the attached figure.

[0202] Construct the first scene, the scene in the first set of multiple scenes, and the scene in the second set of multiple scenes without rotating the model panel by 140.

[0203] For each i∈{1,2,3}, and for each j∈{1,2,…,10}, position P (i),j In reference frame F (i) The middle can be composed of three coordinates Description. For each i∈{1,2,3}, the reference frame F (i) It is chosen such that its (X,Y) plane includes ten locations P. (i),1 ,、P (i),2 ... P (i),10 In particular, for each i∈{1,2,3} and for each j∈{1,2,…,10},

[0204] In particular, the reference frame F (1) The coordinates can be transformed into reference frame F by using the fourth and fifth rotations and translations respectively. (2) The coordinates and transformation to reference frame F (3) The coordinates of i. For each i∈{1,2,3}, the reference frame F (i) By using the reference frame F (i) Each axis uses the same unit length L (i) To limit. Especially L (1) =L (2) =L (3) .

[0205] In step 216, DPS160 calibrates camera 150 by using the positions of a first plurality of calibration points 60-69 as positions in the image plane projected onto the image plane of the first calibration view, using the positions of a second plurality of calibration points 90-99 as positions in the image plane projected onto the image plane of the second calibration view, and using the positions of a fourth plurality of calibration points 50-59 as positions in the image plane projected onto the image plane of the third calibration view. Specifically, camera 150 is calibrated by using calibration points M for each j∈{1,2,…,10} and each i∈{1,2,3}. (i),j As to position P (i),j The position of the projection on the image plane is determined by the image plane.

[0206] The calibration of camera 150 is performed by minimizing a function of parameters given below:

[0207]

[0208] In formula (15), for each j∈{1,2,…,10} and each i∈{1,2,3}, q (i),j It is achieved by using formulas (5) to (9) and replacing (1) → (i) and F → F. (i) It is calculated using the rotation matrix R. (2) Translation vector It is parameterized by parameters from a set of parameters associated with the second view. This set of parameters may include the translation vector. The three entries and the rotation matrix R (2) The entry is parameterized by three parameters. A set of parameters associated with the second view consists of A elements, which can be collected in a six-dimensional list. In the middle. Rotation matrix R (3) Translation vector It is parameterized by parameters from a set of parameters associated with the third view. This set of parameters may include the translation vector. The three entries and the rotation matrix R (3) The entries are parameterized using three parameters. Specifically, this set of parameters associated with the third view consists of A elements, which can be collected in a six-dimensional list. middle.

[0209] For those included in the list and The parameters in the table are such that the parametric function g is minimized. If lens distortion is not negligible, then the function g depends on the parameters included in the list k^ via coefficients D(X′,Y′,Z′), see formula (7a). In this case, the function G is for the list... The parameters included are also minimized. The minimization of the function G can be performed using iterative methods such as the Levenberg-Marquart method.

[0210] Alternatively, based on the above, in formula (15), for each j∈{1,2,…,10} and each i∈{1,2,3}, it can be achieved by using formulas (6) to (9), (11) and (12) and by substituting (1)→(i) and F→F. (i) To calculate q (i),j In this case, in particular, the parameter function G can be applied to a list. and Minimize the parameters included in the list. Given from the following:

[0211]

[0212] In a first embodiment of the method of the present invention, the step of constructing the scene in the first plurality of scenes is performed in two sub-steps (steps 204 and 207). Furthermore, the step of acquiring the image in the first plurality of images is also performed in two sub-steps (steps 205 and 208). The step of locating the calibration point in the first plurality of calibration points is also performed in two sub-steps, namely steps 206 and 209.

[0213] According to this embodiment, the step of locating the calibration site among the first plurality of calibration sites begins before the step of acquiring the images among the first plurality of images and before the step of constructing the scenes among the first plurality of scenes, because steps 207 and 208 are performed after step 206.

[0214] Further embodiments of the method according to the invention may include steps 201 to 216 of the first embodiment described above. The preceding embodiments may differ from each other in order and from the first embodiment, according to which steps 201 to 216 are performed. For example, according to one embodiment of the invention, step 207 may be performed after step 204 and before step 205. In conjunction with the above, step 208 may be performed after step 205 and before step 206. For example, in one embodiment of the method, step 206 is performed after step 208 and before step 209.

[0215] Where not explicitly described, various embodiments or aspects and features thereof relating to the accompanying drawings may be combined or interchanged with each other without limiting or expanding the scope of the invention, provided that such combination or interchange is meaningful and in accordance with the meaning of the invention. Advantages described with respect to specific embodiments of the invention or specific figures are also advantages of other embodiments of the invention, as long as applicable.

Claims

1. A computer-implemented method for calibrating an optical recording apparatus by using at least a model panel and a first calibration view, the optical recording apparatus being associated with a field of view and an image plane, the model panel including a set of calibration marks, wherein, The first calibration view includes a first set of calibration positions for the field of view. The method includes at least the following steps: - Construct multiple scenes such that for each position in the first set of calibration positions, a calibration mark in the set of calibration marks is located in the corresponding scene of the multiple scenes at each position, and wherein each of the multiple scenes is constructed by translating the model panel to a corresponding position relative to the optical recording device; - Acquire multiple images, such that each of the multiple scenes is displayed in the corresponding image of the multiple images; - Locate multiple calibration sites such that for each of the first set of calibration locations, the corresponding site of the multiple calibration sites is included in the corresponding image of the multiple images, the corresponding image displays the corresponding calibration mark, and the corresponding calibration mark is located at each location and is displayed in the corresponding site; - The optical recording device is calibrated by using at least one of the plurality of calibration sites as positions in the image plane projected onto the image plane of the first calibration view, wherein the optical recording device is calibrated using a second calibration view, the second calibration view including a second set of calibration positions of the field of view, wherein each position in the second set of calibration positions can be mapped to a corresponding position in the first set of calibration positions using a first rotational translation; and - By using the first rotation and translation, each position in the second set of calibration positions is mapped to the corresponding position in the field of view, thereby constructing the first set of calibration positions.

2. The method according to claim 1, wherein, The positions in the first set of calibration positions are arranged on a first planar region of the field of view, and the positions in the second set of calibration positions are arranged on a second planar region of the field of view, wherein the first planar region and the second planar region form an angle with each other.

3. The method according to claim 2, wherein, The first planar region and the second planar region are substantially perpendicular to each other.

4. The method according to claim 1, wherein, The positions in the first set of calibration positions and the positions in the second set of calibration positions are arranged on the third plane region of the field of view.

5. The method according to any one of claims 1-4, wherein, The positions in the first set of calibration positions are arranged relative to each other such that each position in the first set of calibration positions is located at a corresponding node of the grid diagram.

6. The method according to claim 5, wherein, The grid diagram is a rectangular grid diagram and / or a square grid diagram.

7. The method according to any one of claims 1-4, wherein, The first set of calibration positions includes at least a first position and a second position, which are aligned with each other on a first line at a first distance, and the set of calibration marks includes at least a first mark and a second mark, which are aligned with each other on a second line at a second distance. Wherein, the first distance and the second distance are substantially equal to each other, and the first line and the second line are substantially parallel to each other. Furthermore, the step of constructing the multiple scenes includes constructing a first scene by means of the following steps: translating the model panel to a first position relative to the optical recording device, such that at the first position, the first mark and the second mark are located at the first position and the second position, respectively.

8. The method according to claim 7, wherein, The first set of calibration positions includes at least the third and fourth positions. Wherein, the third position and the fourth position are aligned on a third line, which is substantially parallel to the first line; wherein the first position and the third position are aligned on a fourth line; and the second position and the fourth position are aligned on a fifth line, which is substantially parallel to each other. The step of constructing the multiple scenes includes constructing a second scene by: translating the model panel from the first position to a second position relative to the optical recording device, such that at the second position, the first mark and the second mark are located at the third position and the fourth position, respectively.

9. The method according to any one of claims 1-4, wherein, The set of calibration marks consists of a third mark.

10. The method according to claim 9, wherein, The third marker is a point or a polygon.

11. The method according to claim 9, wherein, The third mark is a square.

12. The method according to any one of claims 1-4, wherein, The model panel includes a plurality of squares forming a square grid or a checkerboard pattern, the square grid or the checkerboard pattern including a set of vertices of the plurality of squares, wherein each calibration mark in the set of calibration marks is a corresponding vertex of the set of vertices or located at a corresponding vertex of the set of vertices.

13. The method according to any one of claims 1-4, wherein, The step of locating the plurality of calibration sites includes locating at least one of the plurality of calibration sites using an image recognition algorithm.

14. The method according to any one of claims 1-4, wherein, The step of calibrating the optical recording device is performed using a set of parameters, which includes parameters for parameterizing the second rotational translation and / or homography transformation.

15. The method according to claim 14, wherein, The step of calibrating the optical recording device includes minimizing a parameter function relative to a parameter in the set of parameters, wherein the parameter function depends on a parameter in the set of parameters.

16. A data processing system, the data processing system comprising: An optical recording apparatus associated with a field of view and an image plane, a translation device for translating a model panel in the field of view, and a processing apparatus configured to perform the method according to any one of claims 1 to 15.

17. An automated laboratory system, the automated laboratory system comprising: An optical recording device associated with a field of view and an image plane, a translation device for translating a model panel in the field of view, a processing device configured to perform the method according to any one of claims 1 to 15, and a worktable for positioning laboratory items.

18. A computer program product comprising instructions that, when executed by a system according to claim 16 or 17, cause the system to perform the method according to any one of claims 1 to 15.

Images