Data processing method and device applied to hand-eye calibration system, equipment and medium

Abstract

The application discloses a data processing method and device applied to a hand-eye calibration system, equipment and a medium. The method comprises the following steps: determining the pose change data of a camera between the relative positions of sampling points in the camera space and the pose change data of a tracker between the relative positions of sampling points in the tracker space, wherein the pose change data comprises displacement change data and rotation change data; determining the rotation offset data of the camera relative to the tracker in the tracker space according to the rotation change data of the camera between the relative positions of sampling points and the rotation change data of the tracker between the relative positions of sampling points; and determining the displacement scaling parameter of the camera relative to the tracker according to the rotation offset data of the camera relative to the tracker, the pose change data of the tracker between the relative positions of sampling points and the displacement change data of the camera between the relative positions of sampling points. The application can determine the displacement scaling relationship between the camera and the tracker.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data processing method and device applied to a hand-eye calibration system, equipment and a medium. BACKGROUND

[0002] With the rapid development of computer technology and digital graphics processing, machine vision has a wide range of applications in scientific research, industrial production and other fields. In robot vision applications, hand-eye calibration is a very basic technology. Among them, the hand-eye calibration method can determine that there is no displacement scaling relationship between the tracker and the camera when the tracker and the real size are the same size, and cannot determine the displacement scaling relationship between the tracker and the camera when the tracker and the real size are not the same size. SUMMARY

[0003] To solve the above technical problems, the embodiments of the present application provide a data processing method and device applied to a hand-eye calibration system, electronic equipment, computer readable storage medium and computer program product.

[0004] According to an aspect of an embodiment of the present application, a data processing method applied to a hand-eye calibration system is provided, the hand-eye calibration system comprising a camera arranged on a tracker, the data processing method applied to the hand-eye calibration system comprising: determining pose change data of the camera between relative positions of sampling points in a camera space and pose change data of the tracker between the relative positions of the sampling points in a tracker space, the pose change data comprising displacement change data and rotation change data; determining rotation offset data of the camera relative to the tracker in the tracker space according to the rotation change data of the camera between the relative positions of the sampling points and the rotation change data of the tracker between the relative positions of the sampling points; and determining displacement scaling parameters of the camera relative to the tracker according to the rotation offset data of the camera relative to the tracker, the pose change data of the tracker between the relative positions of the sampling points and the displacement change data of the camera between the relative positions of the sampling points.

[0005] According to an aspect of the embodiments of the present application, a data processing device applied to a hand-eye calibration system is provided, the device comprising: a pose change determination module configured to determine pose change data of a camera between relative positions of sampling points in a camera space and pose change data of a tracker between the relative positions of the sampling points in a tracker space, the pose change data comprising displacement change data and rotation change data; a rotation offset determination module configured to determine rotation offset data of the camera relative to the tracker in the tracker space according to the rotation change data of the camera between the relative positions of the sampling points and the rotation change data of the tracker between the relative positions of the sampling points; and a displacement scaling parameter determination module configured to determine displacement scaling parameters of the camera relative to the tracker according to the rotation offset data of the camera relative to the tracker, the pose change data of the tracker between the relative positions of the sampling points and the displacement change data of the camera between the relative positions of the sampling points.

[0006] According to an aspect of the embodiments of the present application, an electronic device is provided, comprising: a memory storing computer readable instructions; and a processor reading the computer readable instructions stored in the memory to execute the data processing method applied to the hand-eye calibration system.

[0007] According to an aspect of the embodiments of the present application, a computer readable storage medium storing computer readable instructions is provided, when the computer readable instructions are executed by a processor of a computer, the computer executes the data processing method applied to the hand-eye calibration system.

[0008] According to an aspect of the embodiments of the present application, a computer program product is also provided, comprising a computer program, when the computer program is executed by a processor, the steps of the data processing method applied to the hand-eye calibration system are implemented.

[0009] In the technical solutions provided by the embodiments of the present application, compared with the current hand-eye calibration method which cannot determine the displacement scaling relationship between the tracker and the camera when the tracker and the real size are not of the same size, the present application can determine the rotation offset data of the camera relative to the tracker in the tracker space according to the rotation change data of the camera between the relative positions of the sampling points and the rotation change data of the tracker between the relative positions of the sampling points, and determine the displacement scaling parameters of the camera relative to the tracker according to the rotation offset data of the camera relative to the tracker, the pose change data of the tracker between the relative positions of the sampling points and the displacement change data of the camera between the relative positions of the sampling points, thereby improving the hand-eye calibration method and realizing the determination of the displacement scaling parameters between the tracker and the camera when the tracker and the real size are not of the same size.

[0010] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application, as claimed. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and serve to explain the principles of the application. It is readily apparent to one skilled in the art that the following figures are merely some embodiments of the present application, and other figures can be obtained by one of ordinary skill in the art without any creative work on the premise that the following description of the figures is merely some embodiments of the present application. In the drawings:

[0012] Figure 1 is a schematic diagram of an implementation environment of a data processing method applied to a hand-eye calibration system according to an exemplary embodiment of the present application;

[0013] Figure 2 is a flowchart of a data processing method applied to a hand-eye calibration system according to an exemplary embodiment of the present application;

[0014] Figure 3 is a flowchart of a data processing method applied to a hand-eye calibration system according to another exemplary embodiment of the present application;

[0015] Figure 4 is a flowchart of a data processing method applied to a hand-eye calibration system according to another exemplary embodiment of the present application;

[0016] Figure 5 is a flowchart of a data processing method applied to a hand-eye calibration system according to another exemplary embodiment of the present application;

[0017] Figure 6 is a flowchart of a data processing method applied to a hand-eye calibration system according to another exemplary embodiment of the present application;

[0018] Figure 7 is a flowchart of a data processing method applied to a hand-eye calibration system according to another exemplary embodiment of the present application;

[0019] Figure 8 is a schematic diagram of a calibration board in a data processing method applied to a hand-eye calibration system according to an exemplary embodiment of the present application;

[0020] Figure 9 is a schematic diagram of a calibration board in a data processing method applied to a hand-eye calibration system according to another exemplary embodiment of the present application;

[0021] Figure 10is a flow chart of a data processing method applied to a hand-eye calibration system according to another example embodiment of the present application;

[0022] Figure 11 is a flow chart of a data processing method applied to a hand-eye calibration system according to another example embodiment of the present application;

[0023] Figure 12 is a flow chart of a data processing method applied to a hand-eye calibration system according to another example embodiment of the present application;

[0024] Figure 13 is a flow chart of a data processing method applied to a hand-eye calibration system according to another example embodiment of the present application;

[0025] Figure 14 is a flow chart of a data processing method applied to a hand-eye calibration system according to another example embodiment of the present application;

[0026] Figure 15 is a flow chart of a data processing method applied to a hand-eye calibration system according to another example embodiment of the present application;

[0027] Figure 16 is a flow chart of a data processing method applied to a hand-eye calibration system according to another example embodiment of the present application;

[0028] Figure 17 is a structural schematic diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0029] The example embodiments will be described in detail herein with reference to the drawings. When the following description refers to arrangements in the drawings, identical numbers on different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following example embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples consistent with some aspects of the present application as detailed in the appended claims.

[0030] The block diagrams shown in the drawings are merely functional entities, and do not necessarily have to correspond to physically independent entities. That is, the functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0031] The flowcharts shown in the drawings are only illustrative, and do not necessarily include all contents and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be further decomposed, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to actual conditions.

[0032] "Multiple" mentioned in the present application refers to two or more. "And / or" describes the association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can represent three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the front and rear associated objects are in an "or" relationship.

[0033] First of all, it needs to be pointed out that with the rapid development of computer technology and digital graphics processing, machine vision has a wide range of applications in scientific research, industrial production and other fields. Among them, the robot in machine vision also refers to a multi-joint multi-degree-of-freedom mechanical arm. The mechanical arm is driven by a rotating motor to achieve controllable positioning of the robot end. It needs to be pointed out that the robot itself does not have a sensor, so a camera or a video camera can be artificially installed on or beside the robot. The target coordinates are obtained by using the camera or the video camera, so that the robot can operate the target according to the image obtained by the camera or the video camera. The way is called robot vision. In order to establish a relationship between the camera or the video camera (i.e. the eyes of the robot) and the coordinate system of the robot (i.e. the hands of the robot), the coordinate system of the robot and the camera or the camera needs to be calibrated. This calibration process can be called hand-eye calibration. The current hand-eye calibration method cannot determine the displacement scaling relationship between the tracker and the camera when the tracker and the real size are not the same size.

[0034] Based on this, in order to determine the displacement scaling relationship between the tracker and the camera when the tracker and the real size are not the same size, the embodiments of the present application propose a data processing method and device applied to a hand-eye calibration system, electronic equipment, computer readable storage medium and computer program product. Specifically: determine the attitude change data between the relative positions of the camera at the sampling points in the camera space and the attitude change data between the relative positions of the tracker at the sampling points in the tracker space. The attitude change data includes displacement change data and rotation change data; determine the rotation offset data of the camera relative to the tracker in the tracker space according to the rotation change data between the relative positions of the camera at the sampling points and the rotation change data between the relative positions of the tracker at the sampling points; determine the displacement scaling parameter of the camera relative to the tracker according to the rotation offset data of the camera relative to the tracker, the attitude change data between the relative positions of the tracker at the sampling points and the displacement change data between the relative positions of the camera at the sampling points. Thus, by improving the hand-eye calibration method, the displacement scaling parameter between the tracker and the camera is determined when the tracker and the real size are not the same size.

[0035] It should be noted that the hand-eye calibration system includes a tracker and a camera arranged on the tracker, the hand-eye calibration system controls the camera to shoot multi-angle calibration board images, and according to the calibration board in the shot calibration board image, the tracker in the tracker space and the camera in the camera space are established to establish a conversion relationship, so that the hand-eye calibration system controls the robot (also referred to as the hand of the robot) to operate the calibration board according to the camera (also referred to as the eye of the robot).

[0036] Figure 1 It is an exemplary embodiment of the application of the data processing method for the hand-eye calibration system. The implementation environment diagram of the embodiment is shown. The implementation environment includes a hand-eye calibration system 110 and a server 120, and the hand-eye calibration system 110 and the server 120 are pre-established wired or wireless network connection.

[0037] As shown in Figure 1 In the data processing process of the hand-eye calibration system, the camera in the hand-eye calibration system 110 can obtain the related data between the relative positions of the sampling points, and the tracker in the hand-eye calibration system 110 can obtain the related data between the relative positions of the sampling points. The hand-eye calibration system 110 sends the related data of the camera at the relative positions of the sampling points and the related data of the tracker at the relative positions of the sampling points to the server 120. The server determines the attitude change data of the camera in the camera space between the relative positions of the sampling points and the attitude change data of the tracker in the tracker space between the relative positions of the sampling points according to the received related data, and determines the rotation offset data of the camera relative to the tracker in the tracker space according to the rotation change data between the relative positions of the sampling points of the camera and the rotation change data between the relative positions of the sampling points of the tracker. Then, according to the rotation offset data of the camera relative to the tracker, the attitude change data between the relative positions of the sampling points of the tracker, and the displacement change data between the relative positions of the sampling points of the camera, the displacement scaling parameter of the camera relative to the tracker is determined. When the tracker and the real size are not the same size, the displacement offset data of the camera relative to the tracker is accurately determined according to the displacement scaling parameter of the camera relative to the tracker, and the accuracy of the displacement offset data is improved.

[0038] Among them, Figure 1 The tracker in the hand-eye calibration system 110 shown in the figure can be the end of the mechanical arm of the robot, which is used to operate the calibration board; the camera in the hand-eye calibration system 110 is used to shoot calibration board images containing calibration boards from multiple angles. The camera can be a camera, a mobile phone or any terminal device supporting image shooting, but is not limited thereto. Figure 1The server 120 shown is a server, which can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. No restrictions are placed on this. The hand-eye calibration system 110 can communicate with the server 120 via wireless networks such as 3G (third-generation mobile information technology), 4G (fourth-generation mobile information technology), and 5G (fifth-generation mobile information technology). No restrictions are placed on this as well.

[0039] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data processing method applied to a hand-eye calibration system, as shown in an exemplary embodiment of this application. The method can be applied to... Figure 1 The implementation environment shown is specifically executed by the hand-eye calibration system 110 within that implementation environment. It should be understood that this method can also be applied to other exemplary implementation environments and executed by devices in other implementation environments; this embodiment does not limit the implementation environment to which the method is applicable.

[0040] The following section will use a hand-eye calibration system as the specific execution subject to provide a detailed description of the data processing method proposed in this application for a hand-eye calibration system.

[0041] like Figure 2 As shown, in an exemplary embodiment, the data processing method applied to the hand-eye calibration system includes at least steps S210 to S230, which are described in detail below:

[0042] Step S210: Determine the attitude change data of the camera between the relative positions of the sampling points in the camera space and the attitude change data of the tracker between the relative positions of the sampling points in the tracker space.

[0043] First, it should be noted that the relative position of sampling points refers to the relative position of two sampling points during the process of the hand-eye calibration system controlling the tracker to move from one sampling point position to another. Specifically, the first position the tracker moves from its initial position can be one sampling point position, and the second position it moves from that first position can be another sampling point position. It should be noted that the initial position can be a preset position of the tracker within the hand-eye calibration system.

[0044] Camera space refers to the coordinate system in which the camera is located. Tracker space refers to the coordinate system in which the tracker is located.

[0045] The pose change data includes displacement change data and rotation change data. For example, the pose change data can be the displacement change of the tracker in the tracker space when the tracker moves from one sampling point to another sampling point, or the rotation change of the tracker in the tracker space when the tracker moves from one sampling point to another sampling point, or the rotation change of the camera in the camera space when the camera moves from one sampling point to another sampling point, or the displacement change of the camera in the camera space when the camera moves from one sampling point to another sampling point.

[0046] When the hand-eye calibration system controls the tracker to move from one sampling point to another sampling point, the camera on the tracker moves with the tracker. During the movement, the sensor in the hand-eye calibration system determines the pose change data of the tracker in the tracker space and determines the pose change data of the camera in the camera space.

[0047] In step S220, the rotation offset data of the camera relative to the tracker in the tracker space is determined according to the rotation change data of the camera between the relative positions of the sampling points and the rotation change data of the tracker between the relative positions of the sampling points.

[0048] The rotation offset data of the camera relative to the tracker in the tracker space refers to the rotation change of the tracker and the camera between the relative positions of the sampling points.

[0049] The hand-eye calibration system determines the rotation offset data of the camera relative to the tracker in the tracker space according to the rotation change data of the tracker when the tracker moves from one sampling point to another sampling point and the rotation change data of the camera when the camera moves with the tracker.

[0050] In step S230, the displacement scaling parameter of the camera relative to the tracker is determined according to the rotation offset data of the camera relative to the tracker, the pose change data of the tracker between the relative positions of the sampling points, and the displacement change data of the camera between the relative positions of the sampling points.

[0051] The displacement scaling parameter of the camera relative to the tracker refers to the displacement relationship of the tracker relative to the camera that moves with the tracker when the tracker moves from one sampling point to another sampling point.

[0052] The hand-eye calibration system determines the displacement scaling parameter of the camera relative to the tracker according to the posture change data of the tracker during movement from one sampling point position to another sampling point position, the displacement change data of the camera as the tracker moves, and the rotational offset data of the camera relative to the tracker in the tracker space.

[0053] It can be seen that, compared with the current hand-eye calibration method, the data processing method applied to the hand-eye calibration system in the embodiment can determine the displacement scaling parameter between the tracker and the camera when the tracker and the real size are not of the same size, according to the rotational change data of the camera between the relative positions of the sampling points and the rotational change data of the tracker between the relative positions of the sampling points, determine the rotational offset data of the camera relative to the tracker in the tracker space according to the rotational offset data of the camera relative to the tracker, the posture change data of the tracker between the relative positions of the sampling points, and the displacement change data of the camera between the relative positions of the sampling points, and determine the displacement scaling parameter of the camera relative to the tracker, thereby improving the hand-eye calibration method and realizing the determination of the displacement scaling parameter between the tracker and the camera when the tracker and the real size are not of the same size.

[0054] Figure 3 is a flowchart of a data processing method applied to a hand-eye calibration system according to another exemplary embodiment of the present application. In Figure 3 In the embodiment shown, the displacement scaling parameter of the camera relative to the tracker is determined according to the calculated intermediate matrix, and other steps of data processing are described in the foregoing embodiments and will not be described here.

[0055] As shown in Figure 3 In an exemplary embodiment, the process of determining the displacement scaling parameter of the camera relative to the tracker according to the rotational offset data of the camera relative to the tracker, the posture change data of the tracker between the relative positions of the sampling points, and the displacement change data of the camera between the relative positions of the sampling points in step S230 at least further includes steps S310 to S320, which are described in detail as follows:

[0056] Step S310: An intermediate matrix is calculated according to the rotational offset data of the camera relative to the tracker, the displacement change data of the camera between the relative positions of the sampling points, and the rotational change data of the tracker between the relative positions of the sampling points.

[0057] The intermediate matrix is determined based on the rotation offset data of the camera relative to the tracker, the displacement change data of the camera between the relative positions of the sampling points, and the rotation change data of the tracker between the relative positions of the sampling points. It can be understood that the matrix capable of finally determining the displacement scaling parameter of the camera relative to the tracker is determined based on the rotation offset data of the camera relative to the tracker, the displacement change data of the camera between the relative positions of the sampling points, and the rotation change data of the tracker between the relative positions of the sampling points.

[0058] The hand-eye calibration system calculates the intermediate matrix according to the rotation change data of the tracker during movement from one sampling point position to another sampling point position, the displacement change data of the camera moving with the tracker, and the rotation offset data of the camera relative to the tracker in the space of the tracker.

[0059] In step S320, the displacement scaling parameter of the camera relative to the tracker is calculated according to the displacement change data of the tracker between the relative positions of the sampling points and the intermediate matrix.

[0060] The hand-eye calibration system calculates the displacement scaling parameter of the camera relative to the tracker according to the displacement change data of the tracker during movement from one sampling point position to another sampling point position and the intermediate matrix.

[0061] Please continue to refer to Figure 4 , Figure 4 is Figure 3 the flow chart of an exemplary embodiment of step S310 in the data processing method applied to the hand-eye calibration system. In Figure 4 the embodiment shown in FIG. 13, for the process of calculating the intermediate matrix in step S310 according to the rotation offset data of the camera relative to the tracker, the displacement change data of the camera between the relative positions of the sampling points, and the rotation change data of the tracker between the relative positions of the sampling points, at least steps S311-S313 are included, which are introduced as follows:

[0062] In step S311, the displacement change matrix of the camera in the real space is determined according to the rotation offset data of the camera relative to the tracker and the displacement change data of the camera between the relative positions of the sampling points.

[0063] The real space refers to the world coordinate system.

[0064] The displacement change matrix of the camera in the real space refers to the displacement change matrix of the camera in the world coordinate system during movement from one sampling point position to another sampling point position.

[0065] The hand-eye calibration system determines the displacement change matrix of the camera in the real space according to the displacement change data of the camera during movement from one sampling point position to another sampling point position and the rotation offset data of the camera relative to the tracker during the movement.

[0066] For example, if the displacement change data of the camera during the movement from one sampling point position to another sampling point position is represented as Tcij, and the rotational offset data of the camera relative to the tracker during the movement is represented as Rcg, then the displacement change matrix of the camera in the real space is represented as Rcg*Tcij. It should be noted that i represents a sampling point position, j represents another sampling point position, c represents the camera, and g represents the tracker.

[0067] In step S312, a difference matrix is determined according to the difference between the rotational change matrix determined according to the rotational change data of the tracker between the relative positions of the sampling points and the unit matrix.

[0068] The difference matrix is determined based on the rotational change matrix of the tracker between the relative positions of the sampling points and the unit matrix.

[0069] The hand-eye calibration system calculates a matrix difference between the displacement change matrix determined according to the displacement change data of the tracker during the movement from one sampling point position to another sampling point position in the tracker space and the unit matrix.

[0070] For example, if the displacement change matrix determined according to the displacement change data of the tracker during the movement from one sampling point position to another sampling point position in the tracker space is represented as Rgij, and the unit matrix is represented as I, then the difference matrix is represented as Rgij-I.

[0071] In step S313, the displacement change matrix and the difference matrix are combined to obtain an intermediate matrix.

[0072] Combining the displacement change matrix and the difference matrix means splicing the displacement change matrix and the difference matrix. For example, if the difference matrix RgiJ-I is represented as Rxij, and the displacement change matrix of the camera in the real space Rcg*Tcij is represented as Txij, then the intermediate matrix is represented as RxijTxij. For example, the difference matrix Rxij is The displacement change matrix of the camera in the real space Txij is Then the intermediate matrix is

[0073] Please continue to refer to Figure 5 , Figure 5 is Figure 3 the flowchart of an exemplary embodiment of step S320 in the data processing method applied to the hand-eye calibration system. In Figure 5 In the embodiment shown in FIG. 8, for the process of step S320 of calculating the displacement scaling parameter of the camera relative to the tracker according to the displacement change data of the tracker between the relative positions of the sampling points and the intermediate matrix, at least steps S321-S323 are further included, which are introduced as follows:

[0074] Step S321, determining the inverse of the displacement variation matrix of the camera between the relative positions of the sampling points according to the displacement variation data of the camera between the relative positions of the sampling points.

[0075] The hand-eye calibration system determines the inverse of the displacement variation matrix of the camera between the relative positions of the sampling points according to the displacement variation data of the camera between the relative positions of the sampling points. For example, if the displacement variation data of the camera between the relative positions of the sampling points is represented as Tgij, then the inverse of the displacement variation matrix of the camera between the relative positions of the sampling points is -Tgij. For example, the inverse of the displacement variation matrix of the camera between the relative positions of the sampling points -Tgij can be

[0076] Step S322, calculating the product between the inverse and the inverse of the intermediate matrix to obtain a first product matrix.

[0077] The first product matrix is determined based on the inverse of the displacement variation matrix of the camera between the relative positions of the sampling points and the inverse of the intermediate matrix.

[0078] The hand-eye calibration system calculates the product between the inverse of the displacement variation matrix of the camera between the relative positions of the sampling points and the inverse of the intermediate matrix to obtain a first product matrix. For example, if the intermediate matrix is The inverse of the intermediate matrix is The inverse of the displacement variation matrix of the camera between the relative positions of the sampling points is Then the first product matrix is

[0079] Step S323, obtaining the displacement scaling parameter of the camera relative to the tracker from the first product matrix.

[0080] The hand-eye calibration system determines the displacement scaling parameter of the camera relative to the tracker according to the first product matrix. Continuing the above example, if the first product matrix is Then S3 in the first product matrix is the displacement scaling parameter of the camera relative to the tracker.

[0081] Optionally, the process can continue to refer to Figure 6 , Figure 6 is Figure 5 the flow chart of an exemplary embodiment of step S323 in the data processing method applied to the hand-eye calibration system shown in FIG. 3. In the embodiment shown in FIG. 3, the process of obtaining the displacement scaling parameter of the camera relative to the tracker from the first product matrix in step S323 further includes steps S3231-S3232, which are described in detail as follows: Figure 6

[0082] ​Step S3231, determining the parameter corresponding to the preset position from the first product matrix.

[0083] The preset position can be a position of an element in the first product matrix except a position of a last element.

[0084] The parameter corresponding to the preset position is an element in the first product matrix.

[0085] The hand-eye calibration system determines the parameter corresponding to the position of the element in the first product matrix except the position of the last element.

[0086] Step S3232, obtaining the parameter corresponding to the preset position as the displacement offset data of the camera relative to the tracker.

[0087] The hand-eye calibration system determines the parameter corresponding to the position of the element in the first product matrix except the position of the last element as the displacement offset data of the camera relative to the tracker. For example, if the first product matrix is the displacement offset data of the camera relative to the tracker is

[0088] As can be seen from the above, the calculation of the displacement offset data of the camera relative to the tracker and the displacement scaling parameter of the embodiment of the application satisfies the following formula:

[0089] (Rgij-I)*Tcg-Rcg*Tcij*Scale=-Tgij

[0090] wherein Rgij refers to the rotation change data of the tracker in the process of moving from one sampling point position to another sampling point position in the tracker space, i.e., a rotation change matrix, I refers to a unit matrix, Tcg refers to the displacement offset data of the tracker relative to the camera in the moving process, Rcg refers to the rotation offset data of the tracker relative to the camera in the moving process, Tcij refers to the displacement change data of the camera in the process of moving from one sampling point position to another sampling point position in the camera space, Tgij refers to the displacement change data of the tracker in the process of moving from one sampling point position to another sampling point position in the tracker space, and Scale refers to the displacement scaling parameter of the camera relative to the tracker.

[0091] It can be seen that the data processing method applied to the hand-eye calibration system according to the embodiment determines the displacement change matrix of the camera in the real space according to the rotation offset data of the camera relative to the tracker and the displacement change data between the relative positions of the sampling points of the camera; determines the difference matrix between the difference value between the rotation change matrix determined according to the rotation change data between the relative positions of the sampling points of the tracker and the unit matrix; combines the displacement change matrix and the difference matrix to obtain an intermediate matrix, and determines the inverse number of the displacement change matrix of the camera between the relative positions of the sampling points according to the displacement change data between the relative positions of the sampling points of the camera; calculates the product between the inverse number and the inverse matrix of the intermediate matrix to obtain a first product matrix; obtains the displacement scaling parameter of the camera relative to the tracker and the displacement offset data of the camera relative to the tracker from the first product matrix. Compared with the displacement offset determined by the existing hand-eye calibration algorithm when the tracker and the real size are not the same size, the present application considers the scaling ratio between the tracker and the real size to satisfy that the accurate displacement offset can be obtained when the tracker and the real size are the same or different, and the scaling ratio between the tracker and the real size can be obtained.

[0092] Figure 7 is another exemplary embodiment of the present application shows a flow chart of a data processing method applied to a hand-eye calibration system. In Figure 7 In the embodiment shown, the camera pose change data between the relative positions of the sampling points of the camera is calculated according to the determined camera intrinsic parameters, and other steps of data processing are described in the foregoing embodiments, which will not be described here.

[0093] As Figure 7 shown, in an exemplary embodiment, the process of determining the camera pose change data between the relative positions of the sampling points of the camera in the camera space and the tracker pose change data between the relative positions of the sampling points of the tracker in the tracker space in step S210 further includes steps S710 to S720, which are described in detail as follows:

[0094] Step S710, calculating the camera intrinsic parameters according to the calibration parameters of the calibration board and the obtained multiple images of the calibration board at different angles.

[0095] Since the calibration board includes a checkerboard, the relationship between the camera and the tracker can be established by the corner points of the calibration board in the calibration board image captured by the camera, that is, the tracker space and the camera space are calibrated. Details can be referred to the calibration board shown in Figure 8 .

[0096] The calibration parameters of the calibration board include the number of horizontal corner points, the number of vertical corner points, the horizontal size of each grid, etc. As Figure 8The calibration board shown has 7 lateral corner points and 5 longitudinal corner points, and each grid has a lateral size of 1850.

[0097] Different angle calibration board images refer to images of the calibration board taken from different angles, details of which are as follows. Figure 8 An image of the calibration board taken at a middle angle, and Figure 9 An image of the calibration board taken at an angle.

[0098] Camera internal parameters refer to parameter settings for internal image shooting of a camera.

[0099] The hand-eye calibration system calculates camera internal parameters using the calibration parameters of the set calibration board and the calibration images containing the calibration board taken from multiple angles. It should be noted that different focal lengths correspond to different camera internal parameters, so when preparing for the calculation of camera internal parameters, the focal length of the camera should be fixed. Of course, if there are multiple focal lengths, corresponding internal parameters can be generated for different focal lengths, and then the corresponding camera internal parameters can be selected according to the focal length in different scenes.

[0100] Step S720, according to the camera internal parameters and multiple calibration board images of different angles, the pose change data of the camera between the relative positions of the sampling points in the camera space is calculated.

[0101] The hand-eye calibration system calculates the pose change data of the camera between the relative positions of the sampling points in the camera space according to the obtained camera internal parameters and the calibration board images taken from different angles.

[0102] For example, please continue to refer to Figure 10 , Figure 10 is Figure 7 the flowchart of an exemplary embodiment of step S710 in the data processing method applied to the hand-eye calibration system. In Figure 10 the embodiment shown, for the process of calculating camera internal parameters according to the calibration parameters of the calibration board and the obtained multiple calibration board images of different angles in step S710, at least steps S711-S713 can be further included, which are introduced as follows:

[0103] Step S711, determine the pixel coordinates of each checkerboard grid corner point of the calibration board in the pixel space in the calibration board pattern.

[0104] The pixel coordinates of the checkerboard grid corner points in the pixel space are obtained based on the detection of the corner points of the checkerboard grid in the shooting image.

[0105] The hand-eye calibration system detects the checkerboard grid corner points of the calibration board in the calibration image and determines the pixel coordinates of each corner point in the pixel space.

[0106] Step S712, determining the physical coordinates of each corner point of the checkerboard in the real space according to the size of the checkerboard in the calibration parameters.

[0107] The size of the checkerboard refers to the area of each checkerboard. For example, as shown in the checkerboard in Figure 8 , the size of each checkerboard is 1850x1850.

[0108] The hand-eye calibration system calculates the physical coordinates of each corner point of the checkerboard in the display space according to the size of the checkerboard in the calibration parameters and the coordinate origin of the display space.

[0109] Step S713, calibrating the pixel coordinates of each corner point and the physical coordinates of the corresponding corner point to obtain the camera intrinsic parameters.

[0110] The hand-eye calibration system calibrates the camera intrinsic parameters according to the obtained pixel coordinates of each corner point and the physical coordinates of the corresponding corner point.

[0111] For example, please refer to Figure 11 , Figure 11 is Figure 7 the flowchart of an exemplary embodiment of step S720 in the data processing method applied to the hand-eye calibration system. In the embodiment shown in Figure 11 , for the process of step S720, calculating the pose change data of the camera in the camera space between the relative positions of the sampling points according to the camera intrinsic parameters and the multiple images of the calibration board at different angles, at least steps S721-S723 can be further included, which are introduced as follows:

[0112] Step S721, determining the pixel coordinate matrix of each corner point according to the pixel coordinates of each corner point in the pixel space, and determining the physical coordinate matrix of each corner point according to the physical coordinates of each corner point in the real space.

[0113] The hand-eye calibration system calculates the pixel coordinate matrix of each corner point according to the pixel coordinates of each corner point in the pixel space, and determines the physical coordinate matrix of each corner point according to the physical coordinates of each corner point in the real space.

[0114] Step S722, calculating the product between the pixel coordinate matrix of each corner point and the inverse matrix of the physical coordinate matrix of the corresponding corner point to obtain a second product matrix.

[0115] The second product matrix is determined based on the product between the pixel coordinate matrix of each corner point and the inverse matrix of the physical coordinate matrix of the corresponding corner point.

[0116] The hand-eye calibration system calculates the product between the pixel coordinate matrix of each corner point and the inverse matrix of the physical coordinate matrix of the corresponding corner point to obtain a second product matrix.

[0117] Step S723, determining the ratio between the second product matrix and the camera intrinsic parameter as the pose change data of the camera in the camera space between the relative positions of the sampling points.

[0118] The hand-eye calibration system calculates the ratio between the second product matrix and the camera intrinsic parameter, and takes the ratio between the second product matrix and the camera intrinsic parameter as the pose change data of the camera in the camera space between the relative positions of the sampling points.

[0119] It can be seen that the data processing method applied to the hand-eye calibration system in the embodiment calculates the pose change data of the camera in the camera space between the relative positions of the sampling points by the determined camera intrinsic parameter and the multi-angle calibration image, so that the hand-eye calibration can be improved based on the pose change data of the camera in the camera space between the relative positions of the sampling points, and the displacement scaling parameter of the camera relative to the tracker is obtained.

[0120] Figure 12 is a flowchart of a data processing method applied to a hand-eye calibration system according to another exemplary embodiment of the present application. As shown in Figure 12 , in an exemplary embodiment, the process of determining the rotational offset data of the camera relative to the tracker in the tracker space according to the rotational change data of the camera between the relative positions of the sampling points and the rotational change data of the tracker between the relative positions of the sampling points in step S220 at least further includes steps S1210 to S1220, which are described in detail as follows:

[0121] Step S1210, converting the rotational change matrix determined by the rotational change data of the camera between the relative positions of the sampling points into a first rotation vector, and converting the rotational change matrix determined by the rotational change data of the tracker between the relative positions of the sampling points into a second rotation vector.

[0122] The first rotation vector is obtained by converting the rotational change matrix of the camera between the relative positions of the sampling points by using Rodrigues.

[0123] The second rotation vector is obtained by converting the rotational change matrix of the tracker between the relative positions of the sampling points by using Rodrigues.

[0124] For example, the first rotation vector can be expressed as r gij , where r gij = rodrigues(R gij ). The second rotation vector is expressed as r cij , where r cij = rodrigues(R cij ).

[0125] The hand-eye calibration system converts a rotation change matrix determined by the camera from rotation change data between relative positions of the sampling points into a first rotation vector, and converts a rotation change matrix determined by the tracker from rotation change data between relative positions of the sampling points into a second rotation vector.

[0126] In step S1220, a rotation matrix is calculated based on the first rotation vector and the second rotation vector to obtain rotation offset data of the camera relative to the tracker in the tracker space.

[0127] The hand-eye calibration system calculates the rotation offset data of the camera relative to the tracker in the tracker space by using the first rotation vector and the second rotation vector.

[0128] For example, please continue to refer to Figure 13 , Figure 13 is Figure 12 the flow chart of an exemplary embodiment of step S1220 in the data processing method applied to the hand-eye calibration system. In Figure 10 the embodiment shown in FIG. 12, for the process of step S1220, the rotation matrix is calculated based on the first rotation vector and the second rotation vector to obtain the rotation offset of the camera relative to the tracker in the tracker space, which can further include steps S1221-S1223, which are described as follows:

[0129] In step S1221, the corresponding pose change representation parameters are determined according to the first rotation vector and the second rotation vector, respectively.

[0130] The pose change representation parameters are determined based on the modified Rodrigues parameters.

[0131] After the hand-eye calibration system performs vector normalization on the first rotation vector, the modified Rodrigues parameters are used to represent the pose change of the first rotation vector to obtain the pose change representation parameters of the first rotation vector, and after performing vector normalization on the second rotation vector, the modified Rodrigues parameters are used to represent the pose change of the second rotation vector to obtain the pose change representation parameters of the second rotation vector.

[0132] In step S1222, the initial rotation vector of the camera relative to the tracker in the tracker space is calculated according to the pose change representation parameters of the first rotation vector and the pose change representation parameters of the second rotation vector.

[0133] The hand-eye calibration system calculates the initial rotation vector of the camera relative to the tracker in the tracker space by using the pose change representation parameters of the first rotation vector and the pose change representation parameters of the second rotation vector. For example, if the pose change representation parameters of the first rotation vector are P gij and the pose change representation parameters of the second rotation vector are P cijThe calculation of the initial rotation vector satisfies the following formula:

[0134] skew(P gij +P cij )P′ cg =P cij -P gij

[0135] P′ cg is the initial rotation vector of the camera relative to the tracker in the tracker space.

[0136] In step S1223, the rotation offset data of the camera relative to the tracker in the tracker space is calculated according to the initial rotation vector.

[0137] The hand-eye calibration system calculates the rotation offset data of the camera relative to the tracker in the tracker space according to the initial rotation vector of the camera relative to the tracker in the tracker space.

[0138] Specifically, the rotation offset data of the camera relative to the tracker in the tracker space satisfies the following formula:

[0139]

[0140] P cg is the rotation offset data of the camera relative to the tracker in the tracker space.

[0141] It can be seen that the data processing method applied to the hand-eye calibration system of the embodiment converts the rotation change matrix determined by the rotation change data of the camera between the relative positions of the sampling points into a first rotation vector, and converts the rotation change matrix determined by the rotation change data of the tracker between the relative positions of the sampling points into a second rotation vector; the rotation matrix is calculated based on the first rotation vector and the second rotation vector, so that the rotation offset of the camera relative to the tracker in the tracker space can be obtained, and the accuracy of the rotation offset is improved.

[0142] In order to describe the data processing method applied to the hand-eye calibration system of the present application in detail, the flow chart shown in FIG. 1 is used to further illustrate it, and the details are as follows: Figure 14

[0143] ​The hand-eye calibration system first performs camera internal parameter preparation, that is, fixes the camera focal length and sets the calibration board parameters, and calculates the camera internal parameters by using the calibration board parameters and the obtained multiple calibration board images at different angles. Then, the displacement offset data and the displacement scaling parameters between the camera and the tracker are calculated. Specifically, the hand-eye calibration system calculates the attitude change data of the camera at the sampling points in the camera space according to the calculated camera internal parameters and the multiple calibration board images at different angles, and calculates the attitude change data of the tracker at the sampling points in the tracker space according to the SDK record in the tracker. Then, the current hand-eye calibration algorithm is improved according to the attitude change data of the camera at the sampling points and the attitude change data of the tracker at the sampling points. Then, the displacement offset data and the displacement scaling parameters of the camera relative to the tracker are determined based on the improved hand-eye calibration algorithm, so that the accuracy of the displacement offset data of the camera relative to the tracker is improved, and the displacement scaling parameters between the tracker and the camera can be determined when the tracker and the real size are not the same size.

[0144] It should be noted that the data processing method applied to the hand-eye calibration system disclosed in the embodiments of the present application can be applied to a virtual production scene. The virtual production refers to a data-based workflow of computer-aided production and film visualization production. With the promotion of real-time rendering technology, its applications include pre-production virtual rehearsal, real-time motion capture of virtual characters, green screen virtual production, and LED virtual production. The effect of the data processing method applied to the hand-eye calibration system in the LED virtual production scene is shown in FIG. 1. Figure 15

[0145] Please refer to Figure 15 , Figure 15 The virtual production scene includes an LED screen wall, a person to be photographed, and a hand-eye calibration system. The hand-eye calibration system determines a virtual background containing the posture of the person to be photographed in the virtual camera according to the calculated displacement scaling parameters of the camera relative to the tracker and the displacement offset data of the camera relative to the tracker, and maps the virtual background containing the posture of the person to be photographed to the LED screen wall, so that the real camera can directly photograph the person to be photographed and determine the virtual background information. That is, it can be understood that in the virtual production scene, the displacement scaling parameters of the camera relative to the tracker and the displacement offset data of the camera relative to the tracker determined by the data processing method applied to the hand-eye calibration system of the present application can improve the consistency of the camera shooting picture and the picture displayed on the LED screen wall. If the displacement scaling parameters of the camera relative to the tracker and the displacement offset data of the camera relative to the tracker determined by the data processing method applied to the hand-eye calibration system of the present application are incorrect, the camera shooting picture and the picture displayed on the LED screen wall will be inconsistent. ​

[0146] Figure 16 is a block diagram of a data processing apparatus applied to a hand-eye calibration system according to an example embodiment of the present application. The data processing apparatus applied to the hand-eye calibration system can be applied to the implementation environment shown in Figure 1 . The data processing apparatus applied to the hand-eye calibration system can also be applicable to other example implementation environments and be configured in other devices, and the implementation environment to which the apparatus is applicable is not limited by the present embodiment.

[0147] As shown in Figure 16 , the example data processing apparatus applied to the hand-eye calibration system 1600 includes a pose change determination module 1610, a rotation offset determination module 1620, and a displacement scaling parameter determination module 1630. Specifically:

[0148] The pose change determination module 1610 is configured to determine pose change data of the camera in the camera space between the relative positions of the sampling points and pose change data of the tracker in the tracker space between the relative positions of the sampling points, the pose change data including displacement change data and rotation change data.

[0149] The rotation offset determination module 1620 is configured to determine rotation offset data of the camera relative to the tracker in the tracker space according to the rotation change data of the camera between the relative positions of the sampling points and the rotation change data of the tracker between the relative positions of the sampling points.

[0150] The displacement scaling parameter determination module 1630 is configured to determine displacement scaling parameters of the camera relative to the tracker according to the rotation offset data of the camera relative to the tracker, the pose change data of the tracker between the relative positions of the sampling points, and the displacement change data of the camera between the relative positions of the sampling points.

[0151] In the example data processing apparatus applied to the hand-eye calibration system, compared with the current hand-eye calibration method which cannot determine the displacement scaling relationship between the tracker and the camera when the tracker and the real size are not the same size, the present application can determine the rotation offset data of the camera relative to the tracker in the tracker space according to the rotation change data of the camera between the relative positions of the sampling points and the rotation change data of the tracker between the relative positions of the sampling points, and determine the displacement scaling parameters of the camera relative to the tracker according to the rotation offset data of the camera relative to the tracker, the pose change data of the tracker between the relative positions of the sampling points, and the displacement change data of the camera between the relative positions of the sampling points, thereby achieving the determination of the displacement scaling parameters between the tracker and the camera when the tracker and the real size are not the same size by improving the hand-eye calibration method.

[0152] On the basis of the above exemplary embodiments, the rotation offset determination module 1620 further comprises an intermediate matrix calculation module and a first calculation submodule. Specifically:

[0153] The intermediate matrix calculation module is configured to calculate an intermediate matrix according to the rotation offset data of the camera relative to the tracker, the displacement change data of the camera between the sampling point relative positions, and the rotation change data of the tracker between the sampling point relative positions.

[0154] The first calculation submodule is configured to calculate the displacement scaling parameter of the camera relative to the tracker according to the displacement change data of the tracker between the sampling point relative positions and the intermediate matrix.

[0155] On the basis of the above exemplary embodiments, the intermediate matrix calculation module further comprises a displacement change matrix determination module, a difference matrix determination module, and a combination module. Specifically:

[0156] The displacement change matrix determination module is configured to determine the displacement change matrix of the camera in the real space according to the rotation offset data of the camera relative to the tracker and the displacement change data of the camera between the sampling point relative positions.

[0157] The difference matrix determination module is configured to determine the difference matrix between the rotation change matrix determined according to the rotation change data of the tracker between the sampling point relative positions and the unit matrix.

[0158] The combination module is configured to combine the displacement change matrix and the difference matrix to obtain the intermediate matrix.

[0159] On the basis of the above exemplary embodiments, the first calculation submodule further comprises an inverse number calculation module, a first product matrix calculation module, and a second calculation submodule. Specifically:

[0160] The inverse number calculation module is configured to determine the inverse number of the displacement change matrix of the camera between the sampling point relative positions according to the displacement change data of the camera between the sampling point relative positions.

[0161] The first product matrix calculation module is configured to calculate the product between the inverse number and the inverse matrix of the intermediate matrix to obtain a first product matrix.

[0162] The second calculation submodule is configured to obtain the displacement scaling parameter of the camera relative to the tracker from the first product matrix.

[0163] On the basis of the above exemplary embodiments, the second calculation submodule further comprises a parameter determination module and a third calculation submodule. Specifically:

[0164] The parameter determination module is configured to determine the parameter corresponding to the preset position from the first product matrix.

[0165] The third sub-computing module is configured to obtain the parameter corresponding to the preset position as displacement offset data of the camera relative to the tracker.

[0166] On the basis of the above exemplary embodiments, the pose change data determination module 1610 further comprises a camera intrinsic parameter computing module and a fourth sub-computing module. Specifically:

[0167] The camera intrinsic parameter computing module is configured to compute the camera intrinsic parameter according to the calibration parameter of the calibration board and the obtained multiple images of the calibration board at different angles.

[0168] The fourth sub-computing module is configured to compute the pose change data of the camera at the sampling points in the camera space according to the camera intrinsic parameter and the multiple images of the calibration board at different angles.

[0169] On the basis of the above exemplary embodiments, the camera intrinsic parameter computing module further comprises an angle point pixel coordinate determining module, an angle point physical coordinate determining module and a calibration module. Specifically:

[0170] The angle point pixel coordinate determining module is configured to determine the pixel coordinates of the angle points of each checkerboard of the calibration board in the pixel space in the calibration board pattern.

[0171] The angle point physical coordinate determining module is configured to determine the physical coordinates of the angle points of each checkerboard in the real space according to the size of the checkerboard in the calibration parameter.

[0172] The calibration module is configured to calibrate the pixel coordinates of each angle point and the physical coordinates of the corresponding angle point to obtain the camera intrinsic parameter.

[0173] On the basis of the above exemplary embodiments, the fourth sub-computing module further comprises a coordinate matrix determining module, a second product matrix determining module and a fifth sub-computing module. Specifically:

[0174] The coordinate matrix determining module is configured to determine the pixel coordinate matrix of each angle point according to the pixel coordinates of each angle point in the pixel space, and determine the physical coordinate matrix of each angle point according to the physical coordinates of each angle point in the real space.

[0175] The second product matrix determining module is configured to compute the product between the pixel coordinate matrix of each angle point and the inverse matrix of the physical coordinate matrix of the corresponding angle point to obtain the second product matrix.

[0176] The fifth sub-computing module is configured to determine the ratio between the second product matrix and the camera intrinsic parameter as the pose change data of the camera at the sampling points in the camera space.

[0177] On the basis of the above exemplary embodiments, the rotation offset determination module 1620 further comprises a rotation vector determination module and a sixth sub-computation module. Specifically,

[0178] The rotation vector determination module is configured to convert the rotation change matrix determined by the rotation change data of the camera between the relative positions of the sampling points into a first rotation vector, and convert the rotation change matrix determined by the rotation change data of the tracker between the relative positions of the sampling points into a second rotation vector.

[0179] The sixth sub-computation module is configured to perform rotation matrix computation based on the first rotation vector and the second rotation vector to obtain the rotation offset of the camera relative to the tracker in the tracker space.

[0180] On the basis of the above exemplary embodiments, the sixth sub-computation module further comprises a pose change representation parameter determination module, an initial rotation vector computation module, and a seventh sub-computation module. Specifically,

[0181] The pose change representation parameter determination module is configured to determine the corresponding pose change representation parameters according to the first rotation vector and the second rotation vector, respectively.

[0182] The initial rotation vector computation module is configured to compute an initial rotation vector of the camera relative to the tracker in the tracker space according to the pose change representation parameters of the first rotation vector and the pose change representation parameters of the second rotation vector.

[0183] The seventh sub-computation module is configured to compute the rotation offset data of the camera relative to the tracker in the tracker space according to the initial rotation vector.

[0184] It should be noted that the data processing apparatus applied to the hand-eye calibration system provided in the above embodiments and the data processing method applied to the hand-eye calibration system provided in the above embodiments belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be repeated here. The data processing apparatus applied to the hand-eye calibration system provided in the above embodiments can be completed by different functional modules according to the needs in actual application, i.e., the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above, and this is not limited herein.

[0185] Embodiments of the present application also provide an electronic device, comprising: one or more processors; a storage apparatus for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the data processing method applied to the hand-eye calibration system provided in each of the above embodiments.

[0186] Figure 17A structural diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application is shown. It should be noted that Figure 17 The computer system 1700 of the electronic device shown is only an example and should not impose any limitation on the functions and usage range of the embodiments of the present application.

[0187] As shown in Figure 17 The computer system 1700 includes a central processing unit (CPU) 1701, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1702 or programs loaded from a storage section 1708 into a random access memory (RAM) 1703, such as performing the methods described in the above embodiments. Various programs and data required for system operation are also stored in the RAM 1703. The CPU 1701, the ROM 1702, and the RAM 1703 are connected to each other through a bus 1704. An input / output (I / O) interface 1705 is also connected to the bus 1704.

[0188] The following components are connected to the I / O interface 1705: an input section 1706 including a keyboard, a mouse, etc.; an output section 1707 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1708 including a hard disk, etc.; and a communication section 1709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1709 performs communication processing via a network such as the Internet. A drive 1710 is also connected to the I / O interface 1705 as necessary. A removable recording medium 1711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1710 as necessary, so that a computer program read therefrom is installed into the storage section 1708 as necessary.

[0189] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing computer programs for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 1709, and / or installed from the detachable medium 1711. When the computer program is executed by the central processing module (CPU) 1701, various functions defined in the system of the present application are executed.

[0190] It should be noted that the computer readable medium shown in the embodiments of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable signal medium can include a data signal propagating in a baseband or as part of a carrier wave, in which the computer readable computer program is carried. Such a propagated data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium that can send, propagate, or transfer the program for use by or in connection with the instruction execution system, apparatus, or device. The computer program contained on the computer readable medium can be transmitted in any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination of the above.

[0191] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowcharts, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0192] The units described in the embodiments of the present application can be implemented by software, or by hardware, or by a combination of software and hardware. The units described may also be located in a single processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0193] Another aspect of the present application provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the data processing method for a hand-eye calibration system as described above. The computer readable storage medium can be included in the electronic device described in the embodiments above, or can exist separately from the electronic device.

[0194] Another aspect of the present application provides a computer program product or computer program, which includes computer instructions. The computer instructions are stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the data processing method for a hand-eye calibration system provided in the embodiments above.

[0195] The above merely illustrates the preferred exemplary embodiments of the present application, and is not intended to limit the implementation of the present application. Based on the main concept and spirit of the present application, those skilled in the art can easily make corresponding modifications or variations, and the protection scope of the present application should be subject to the protection scope as claimed in the claims.

Claims

1. A data processing method applied to a hand-eye calibration system, characterized in that, The hand-eye calibration system includes a camera mounted on the tracker, and the method includes: Determine the attitude change data of the camera between the relative positions of the sampling points in the camera space and the attitude change data of the tracker between the relative positions of the sampling points in the tracker space, wherein the attitude change data includes displacement change data and rotation change data; Based on the rotational change data of the camera between the relative positions of the sampling points and the rotational change data of the tracker between the relative positions of the sampling points, determine the rotational offset data of the camera relative to the tracker in the tracker space; Based on the rotation offset data of the camera relative to the tracker, the attitude change data of the tracker between the relative positions of the sampling points, and the displacement change data of the camera between the relative positions of the sampling points, the displacement scaling parameter of the camera relative to the tracker is determined. When the tracker and the actual size are not the same, the displacement offset data of the camera relative to the tracker is determined according to the displacement scaling parameter of the camera relative to the tracker.

2. The method according to claim 1, characterized in that, The step of determining the displacement scaling parameter of the camera relative to the tracker based on the rotation offset data of the camera relative to the tracker, the attitude change data of the tracker between the relative positions of the sampling points, and the displacement change data of the camera between the relative positions of the sampling points includes: An intermediate matrix is ​​calculated based on the rotation offset data of the camera relative to the tracker, the displacement change data of the camera between the relative positions of the sampling points, and the rotation change data of the tracker between the relative positions of the sampling points; The displacement scaling parameter of the camera relative to the tracker is calculated based on the displacement change data of the tracker between the relative positions of the sampling points and the intermediate matrix.

3. The method according to claim 2, characterized in that, The step of calculating the intermediate matrix based on the rotation offset data of the camera relative to the tracker, the displacement change data of the camera between the relative positions of the sampling points, and the rotation change data of the tracker between the relative positions of the sampling points includes: The displacement change matrix of the camera in real space is determined based on the rotation offset data of the camera relative to the tracker and the displacement change data of the camera between the relative positions of the sampling points. The difference matrix is ​​determined based on the difference between the rotation change matrix and the identity matrix, which is determined by the tracker based on the rotation change data between the relative positions of the sampling points. The intermediate matrix is ​​obtained by combining the displacement change matrix and the difference matrix.

4. The method according to claim 2, characterized in that, The step of calculating the displacement scaling parameter of the camera relative to the tracker based on the displacement change data of the tracker between the relative positions of the sampling points and the intermediate matrix includes: The negative number of the displacement change matrix of the camera between the relative positions of the sampling points is determined based on the displacement change data of the camera between the relative positions of the sampling points; Calculate the product between the opposite number and the inverse of the intermediate matrix to obtain the first product matrix; The displacement scaling parameters of the camera relative to the tracker are obtained from the first product matrix.

5. The method according to claim 4, characterized in that, The step of obtaining the displacement scaling parameters of the camera relative to the tracker from the first product matrix further includes: Determine the parameters corresponding to the preset positions from the first product matrix; The parameters corresponding to the preset position are obtained as the displacement offset data of the camera relative to the tracker.

6. The method according to claim 1, characterized in that, The step of determining the attitude change data of the camera between the relative positions of the sampling points in the camera space and the attitude change data of the tracker between the relative positions of the sampling points in the tracker space, wherein the attitude change data includes displacement change data and rotation change data, includes: The camera intrinsic parameters are calculated based on the calibration parameters of the calibration board and multiple images of the calibration board from different angles. Based on the camera's intrinsic parameters and the multiple calibration board images from different angles, the attitude change data of the camera in the camera space between the relative positions of the sampling points is calculated.

7. The method according to claim 6, characterized in that, The step of calculating the camera intrinsic parameters based on the calibration parameters of the calibration board and multiple calibration board images from different angles includes: Determine the pixel coordinates of the corner points of each checkerboard grid in the calibration board graphic in pixel space; The physical coordinates of the corner points of each chessboard grid in real space are determined based on the size of the chessboard grid in the calibration parameters. The pixel coordinates and physical coordinates of each corner point are calibrated to obtain the camera intrinsic parameters.

8. The method according to claim 6, characterized in that, The step of calculating the attitude change data of the camera between the relative positions of the sampling points in the camera space based on the camera intrinsic parameters and the multiple calibration board images at different angles includes: The pixel coordinate matrix of each corner point is determined based on the pixel coordinates of each corner point in pixel space, and the physical coordinate matrix of each corner point is determined based on the physical coordinates of each corner point in real space. Calculate the product between the pixel coordinate matrix of each corner point and the inverse of the corresponding physical coordinate matrix of the corner point to obtain the second product matrix; The ratio between the second product matrix and the camera intrinsic parameters is determined as the attitude change data of the camera between the relative positions of the sampling points in the camera space.

9. The method according to claim 1, characterized in that, The step of determining the rotational offset data of the camera relative to the tracker in the tracker space based on the rotational change data of the camera between the relative positions of the sampling points and the rotational change data of the tracker between the relative positions of the sampling points includes: The rotation transformation matrix determined by the rotation change data of the camera between the relative positions of the sampling points is transformed into a first rotation vector, and the rotation transformation matrix determined by the rotation change data of the tracker between the relative positions of the sampling points is transformed into a second rotation vector. Based on the first rotation vector and the second rotation vector, a rotation matrix is ​​calculated to obtain the rotation offset data of the camera relative to the tracker in the tracker space.

10. The method according to claim 9, characterized in that, The step of calculating the rotation matrix based on the first rotation vector and the second rotation vector to obtain the rotation offset data of the camera relative to the tracker in the tracker space includes: The corresponding attitude change representation parameters are determined based on the first rotation vector and the second rotation vector, respectively. The initial rotation vector of the camera relative to the tracker in the tracker space is calculated based on the attitude change representation parameters of the first rotation vector and the attitude change representation parameters of the second rotation vector. The rotation offset data of the camera relative to the tracker in the tracker space is calculated based on the initial rotation vector.

11. A data processing device for a hand-eye calibration system, characterized in that, The device includes: The attitude change determination module is configured to determine attitude change data of the camera between relative positions of sampling points in camera space and attitude change data of the tracker between relative positions of sampling points in tracker space, wherein the attitude change data includes displacement change data and rotation change data. The rotation offset determination module is configured to determine the rotation offset data of the camera relative to the tracker in the tracker space based on the rotation change data of the camera between the relative positions of the sampling points and the rotation change data of the tracker between the relative positions of the sampling points. The displacement scaling parameter determination module is configured to determine the displacement scaling parameter of the camera relative to the tracker based on the rotation offset data of the camera relative to the tracker, the attitude change data of the tracker between the relative positions of the sampling points, and the displacement change data of the camera between the relative positions of the sampling points. The displacement offset determination module is configured to determine the displacement offset data of the camera relative to the tracker based on the displacement scaling parameters of the camera relative to the tracker when the tracker and the actual size are not the same.

12. An electronic device, characterized in that, include: Memory, which stores computer-readable instructions; A processor reads computer-readable instructions stored in memory to perform the method described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by a computer's processor, cause the computer to perform the method described in any one of claims 1-10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 10.

Images