Extrinsic parameter calibration method, computing device, image acquisition system and storage medium

Abstract

The application provides an external parameter calibration method, a computing device, an image acquisition system and a storage medium, which comprises: acquiring a plurality of images of a target object at different angles; extracting a plurality of feature points of the target object, and determining a first two-dimensional coordinate of each feature point in each image; establishing a first mapping relationship and a second mapping relationship based on a first external parameter between a three-dimensional coordinate system of a specified image acquisition device in the image acquisition system and a world coordinate system of the target object, and a second external parameter between the three-dimensional coordinate system of the specified image acquisition device and other image acquisition devices; for a plurality of feature points in the plurality of images, determining an error between the second two-dimensional coordinate and the corresponding first two-dimensional coordinate, and when the error meets a preset error condition, determining the first external parameter and the second external parameter. The external parameter calibration of the image acquisition device can be completed by using the user face image without the user's awareness.

Description

Technical Field

[0001] This invention relates to the field of holographic imaging technology, and in particular to an external parameter calibration method, a computing device, an image acquisition system, and a storage medium. Background Technology

[0002] In related technologies, VR (Virtual Reality) / AR (Augmented Reality) and holographic remote video communication products mostly contain multiple camera modules, see reference. Figure 1 The stereoscopic display or 3D interactive effects presented by the product rely on various stereoscopic vision-based algorithms within the product. The output accuracy of each algorithm is largely limited by the accuracy of the intrinsic and extrinsic parameters of each camera module. Typically, the camera's intrinsic parameters (i.e., the camera intrinsic parameter matrix and distortion coefficients) remain fixed after the camera leaves the factory. However, the camera's extrinsic parameters (i.e., the transformation matrix between the coordinate systems of each camera module or the transformation matrix between the camera and the specified world coordinate system) usually change due to assembly errors of the camera modules and offsets caused by vibrations during maintenance or transportation. If the camera's extrinsic parameters are not recalibrated at this time, the product's internal algorithms will use inaccurate extrinsic parameters (i.e., factory parameters) for calculations, ultimately affecting the interactive experience of the system.

[0003] Currently, calibrating camera external parameters typically requires professional operators to simultaneously capture multiple images of a calibration plate of specified dimensions at various angles and positions using each camera module (see reference). Figure 2 The image is used to calculate the transformation matrix between the coordinate systems of each camera module using professional calibration tools (such as MATLAB, OpenCV, etc.). This calibration process requires operators to have a certain professional theoretical foundation (e.g., setting a reasonable placement of the calibration board), purchasing specific professional equipment (e.g., calibration board), and a certain amount of work (moving the calibration board). The complex and professional calibration process will greatly reduce the user experience.

[0004] Therefore, a new method for calibrating camera extrinsic parameters is needed to at least solve the above problems. Summary of the Invention

[0005] The main objective of this invention is to provide an external parameter calibration method, a computing device, an image acquisition system, and a storage medium to simplify the external parameter calibration method for image acquisition devices and provide users with a better user experience.

[0006] This invention provides a method for calibrating the extrinsic parameters of an image acquisition device, comprising: acquiring multiple images of a target object from multiple angles, wherein the multiple images are acquired by multiple image acquisition devices of an image acquisition system, and the target object includes a target face; extracting multiple feature points of the target object and determining a first two-dimensional coordinate of each feature point in each image; establishing a first mapping relationship between the three-dimensional coordinate system of a designated image acquisition device in the image acquisition system and the world coordinate system of the target object, and a second extrinsic parameter between the three-dimensional coordinate systems of the designated image acquisition device and other image acquisition devices, based on a first extrinsic parameter between the three-dimensional coordinate system of the target object and the three-dimensional coordinate system of each image acquisition device; establishing a second mapping relationship between the three-dimensional coordinate of the feature point in the three-dimensional coordinate system of each image acquisition device and its second two-dimensional coordinate in the two-dimensional coordinate system of that image acquisition device, based on the first mapping relationship; for multiple feature points in multiple images, determining the error between the second two-dimensional coordinate and the corresponding first two-dimensional coordinate based on the second mapping relationship, and determining the first extrinsic parameter and the second extrinsic parameter when the error meets a preset error condition.

[0007] In one embodiment, acquiring multiple images of a target object from multiple different angles includes: acquiring multiple images of the target object from multiple different angles when the target object is in different poses; for multiple feature points in the multiple images, determining the error between the second two-dimensional coordinates and the corresponding first two-dimensional coordinates based on the second mapping relationship, and determining the first extrinsic parameter and the second extrinsic parameter when the error meets a preset error condition, including: for multiple feature points in the multiple images of the target object when the target object is in different poses, determining the error between the second two-dimensional coordinates and the corresponding first two-dimensional coordinates based on the second mapping relationship, and determining the first extrinsic parameter and the second extrinsic parameter when the error meets a preset error condition.

[0008] In one embodiment, extracting multiple feature points of a target object and determining the first two-dimensional coordinates of each feature point in each image includes: using an OpenCV algorithm or a machine learning model to extract multiple feature points of the target face and determine the first two-dimensional coordinates of each feature point in each image.

[0009] In one embodiment, based on a first extrinsic parameter between the three-dimensional coordinate system of a designated image acquisition device in the image acquisition system and the world coordinate system of the target object, and a second extrinsic parameter between the three-dimensional coordinate systems of the designated image acquisition device and other image acquisition devices, a first mapping relationship is established between the three-dimensional coordinates of the feature point in the world coordinate system of the target object and its three-dimensional coordinates in the three-dimensional coordinate system of each image acquisition device, including:

[0010] The first mapping relationship is established using the following formula:

[0011]

[0012] in, This represents the three-dimensional coordinates of the feature point in the three-dimensional coordinate system of the i-th image acquisition device when the target object is in the t-th pose, t = 0, 1, ..., n, i = 0, 1, ..., k, ..., m-1, and c represents the image acquisition device. This represents the first extrinsic parameter. i T k The second extrinsic parameter is represented by k, which indicates that the k-th image acquisition device is a specified image acquisition device, and current_shape_3D. t This represents the three-dimensional coordinates of the feature point in the world coordinate system when it is in the t-th pose.

[0013] In one embodiment, based on the first mapping relationship, establishing a second mapping relationship between the three-dimensional coordinates of the feature point in the three-dimensional coordinate system of each image acquisition device and its second two-dimensional coordinates in the two-dimensional coordinate system of the image acquisition device includes:

[0014] The second mapping relationship is established using the following formula:

[0015]

[0016] in, This represents the second two-dimensional coordinate of the j-th feature point in the two-dimensional coordinate system of the i-th image acquisition device when the target object is in the t-th pose, t = 0, 1, ..., n, i = 0, 1, ..., m-1, j = 0, 1, ..., 67, c represents the image acquisition device, K i This represents the intrinsic parameters of the i-th image acquisition device. This represents the three-dimensional coordinates of the j-th feature point in the three-dimensional coordinate system of the i-th image acquisition device when the target object is in the t-th pose.

[0017] In one embodiment, for multiple feature points in multiple images of a target object in different poses, determining the error between the second two-dimensional coordinates and the corresponding first two-dimensional coordinates based on the second mapping relationship includes: determining the error using the following formula:

[0018]

[0019] Wherein, error_All represents the error between the second two-dimensional coordinate and the corresponding first two-dimensional coordinate. This represents the second two-dimensional coordinate of the j-th feature point in the two-dimensional coordinate system of the i-th image acquisition device when the target object is in the t-th pose, t = 0, 1, ..., n, i = 0, 1, ..., m-1, j = 0, 1, ..., 67, c represents the image acquisition device, Landmarks_2d t,i,j This represents the first two-dimensional coordinates of the j-th feature point in the image acquired by the i-th image acquisition device when the target object is in the t-th pose.

[0020] In one embodiment, the designated image acquisition device includes: the image acquisition device most directly facing the target face among the plurality of image acquisition devices; or an image acquisition device whose two-dimensional coordinate plane is parallel to the plane formed by the lines connecting the eyes and mouth of the target face.

[0021] In one embodiment, the method includes: using the least squares method to determine the error that satisfies a preset error condition, and then determining the first extrinsic parameter and the second extrinsic parameter.

[0022] In one embodiment, the method further includes performing distortion correction processing on each image to eliminate image distortion caused by each image acquisition device itself.

[0023] In one embodiment, the distortion includes radial distortion and tangential distortion;

[0024] Radial distortion of the image is corrected using the following formula:

[0025] x0=x(1+k1r 2 +k2r 4 +k3r 6 )

[0026] y0=y(1+k1r 2 +k2r 4 +k3r 6 )

[0027] Where (x0, y0) are the coordinates of any point on the image before correction, (x, y) are the coordinates of any point on the image after correction, and r 2 =x 2 +y 2 k1~k3 represent the radial distortion coefficients;

[0028] The tangential distortion of the image is corrected using the following formula:

[0029] x0=x+[2p1xy+p2(r 2 +2x 2 )]

[0030] y0=x+[2p2xy+p1(r 2 +2y2 )]

[0031] Where (x0, y0) are the coordinates of any point on the image before correction, (x, y) are the coordinates of any point on the image after correction, and p1 and p2 represent the tangential distortion coefficients.

[0032] In one embodiment, the method further includes: determining the three-dimensional coordinates of the feature point in the world coordinate system of the target face using the following formula:

[0033] current_shape_3D t =mean_shape + pv·params t

[0034] Among them, current_shape_3D t This represents the three-dimensional coordinates of the feature point in the world coordinate system of the target face when the target object is in the t-th pose. `mean_shape` represents the preset average face model, `pv` represents the face feature vector, and both `mean_shape` and `pv` are fixed constants. `params` t This represents the iteration parameters when the target object is in the t-th pose.

[0035] The present invention provides a computing device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the above-described external parameter calibration method for an image acquisition device.

[0036] The present invention provides an image acquisition system, including multiple image acquisition devices and the aforementioned computing device.

[0037] The present invention provides a storage medium storing a computer program, which, when executed by the processor, implements the steps of the above-described external parameter calibration method for an image acquisition device.

[0038] Using the method of this invention, images containing the user's face can be captured without the user's awareness. The external parameters of the image acquisition device can be calibrated using the user's face image. The entire process does not require the user's active cooperation or professional operation. It is simple to operate and the user can complete the calibration by himself. Compared with the calibration methods in related technologies, it can bring a better user experience. Attached Figure Description

[0039] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0040] Figure 1 This is a schematic diagram showing the distribution of camera modules in a holographic remote video communication system in related technologies;

[0041] Figure 2 This is a schematic diagram of a calibration board in related technologies;

[0042] Figure 3 This is a flowchart of an external parameter calibration method for an image acquisition device according to an exemplary embodiment of this application;

[0043] Figure 4 This is a schematic diagram of 68 feature points of a target face extracted according to an embodiment of this application. Detailed Implementation

[0044] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0045] During the experience or visit of holographic products, the user's face will mostly appear in the field of view of the various camera modules of the product. Since the face is a typical non-rigid target, the position of the feature points on the face will vary greatly in a natural state. The face can completely replace the calibration plate in the process of calibrating the camera's external parameters.

[0046] Accordingly, the present invention can utilize the facial images of the user in a natural state captured by each camera module when the user uses the product to solve for the optimal solution of the transformation matrix between the coordinate systems of each camera module, i.e., the camera extrinsic parameters, thereby realizing the calibration of the extrinsic parameters of the product's camera modules.

[0047] refer to Figure 3 This embodiment provides a method for calibrating the external parameters of an image acquisition device, including:

[0048] S100: Acquire multiple images of the target object from multiple different angles, wherein the multiple images are acquired by multiple image acquisition devices of the image acquisition system, and the target object includes a target face.

[0049] S200: Extract multiple feature points of the target object and determine the first two-dimensional coordinates of each feature point in each image.

[0050] S300: Based on the first external parameter between the three-dimensional coordinate system of the specified image acquisition device in the image acquisition system and the world coordinate system of the target object, and the second external parameter between the three-dimensional coordinate systems of the specified image acquisition device and other image acquisition devices, establish a first mapping relationship between the three-dimensional coordinates of the feature point in the world coordinate system of the target object and its three-dimensional coordinates in the three-dimensional coordinate system of each image acquisition device.

[0051] S400: Based on the first mapping relationship, establish a second mapping relationship between the three-dimensional coordinates of the feature point in the three-dimensional coordinate system of each image acquisition device and its second two-dimensional coordinates in the two-dimensional coordinate system of the image acquisition device.

[0052] S500: For multiple feature points in multiple images, based on the second mapping relationship, determine the error between the second two-dimensional coordinates and the corresponding first two-dimensional coordinates. When the error meets the preset error condition, determine the first extrinsic parameter and the second extrinsic parameter.

[0053] In this embodiment, the image acquisition device may include devices capable of acquiring images, such as cameras and video cameras; this application does not specifically limit this. The image acquisition system may include multiple image acquisition devices, which may be distributed around the target object. These multiple image acquisition devices can simultaneously acquire images of the target object from different angles, thereby obtaining multiple images of the target object from multiple different angles. (Reference) Figure 1 Among them, 1 to 4 are image acquisition devices, 5 to 8 are synchronous trigger signals, 9 is a synchronization box, and 10 is the target object. When performing image acquisition, the synchronization box can be controlled to send synchronous trigger signals to the four image acquisition devices at the same time, and each image acquisition device can simultaneously acquire multiple images of the target object from different angles.

[0054] In this embodiment, the target object may include a target face; in other embodiments, it may also include rigid objects or other non-rigid objects. When the target object is a face, multiple feature points of the target face can be extracted using the OpenCV algorithm (in OpenCV, an image is a matrix; when reading the pixels of an image, grayscale images directly return grayscale values, while color images return the R, G, and B components (corresponding to the three primary colors red, green, and blue, respectively)). Alternatively, machine learning models can be used to extract multiple feature points of the target face. Examples of machine learning models include dlib (a deep learning-based facial landmark detection algorithm), PFLD (Practical Facial Landmark Detector), etc. The number of feature points can be, for example, 5, 29, 68, 106, or 202, etc., which can be selected according to the needs of those skilled in the art. In this embodiment, 68 feature points can be extracted. Figure 4 .

[0055] In this embodiment, based on the first external parameter between the three-dimensional coordinate system of the specified image acquisition device in the image acquisition system and the world coordinate system of the target object, and the second external parameter between the three-dimensional coordinate systems of the specified image acquisition device and other image acquisition devices, a first mapping relationship is established between the three-dimensional coordinates of the feature point in the world coordinate system of the target object and its three-dimensional coordinates in the three-dimensional coordinate system of each image acquisition device. This allows the three-dimensional coordinates of the feature point in the world coordinate system of the target object to be converted into three-dimensional coordinates in the three-dimensional coordinate system of each image acquisition device.

[0056] In this embodiment, based on the first mapping relationship, a second mapping relationship is established between the three-dimensional coordinates of the feature point in the three-dimensional coordinate system of each image acquisition device and its second two-dimensional coordinates in the two-dimensional coordinate system of the image acquisition device, thereby converting the three-dimensional coordinates of the feature point in each image acquisition device into its two-dimensional coordinates in the image acquisition device.

[0057] In this embodiment, for multiple feature points in multiple images, based on the second mapping relationship, the error between the second two-dimensional coordinates and the corresponding first two-dimensional coordinates is determined. Then, the two-dimensional coordinates obtained by converting the extrinsic parameters of each image acquisition device are compared with the two-dimensional coordinates in the directly acquired image. By controlling the error to meet the preset error conditions, the first extrinsic parameter and the second extrinsic parameter are determined. Furthermore, the determined first and second extrinsic parameters meet the accuracy requirements.

[0058] The extrinsic parameter calibration method of the image acquisition device in this embodiment can acquire images containing the user's face without the user's awareness, and use the user's face image to calibrate the extrinsic parameters of the image acquisition device. The whole process does not require professional operation or user cooperation, and is easy to operate. Users can complete the calibration themselves. Compared with the calibration methods in related technologies, it can bring a better user experience.

[0059] In one embodiment, acquiring multiple images of a target object from multiple different angles may include: acquiring multiple images of the target object from multiple different angles when the target object is in different poses. For multiple feature points in the multiple images, based on the second mapping relationship, determining the error between the second two-dimensional coordinates and the corresponding first two-dimensional coordinates, and determining the first extrinsic parameter and the second extrinsic parameter when the error meets a preset error condition, may include: for multiple feature points in the multiple images of the target object in different poses, determining the error between the second two-dimensional coordinates and the corresponding first two-dimensional coordinates based on the second mapping relationship, and determining the first extrinsic parameter and the second extrinsic parameter when the error meets a preset error condition.

[0060] In this embodiment, when the target object is a target face, multiple images of the target face from multiple different angles in one pose can be acquired at a certain moment without user cooperation, and images of the target face can be acquired at multiple moments, thereby collecting rich image data.

[0061] By acquiring multiple images of the target object at different angles in different postures, the diversity of image data can be increased, making the calibration results of the external parameters of the image acquisition device more accurate.

[0062] In one embodiment, based on a first extrinsic parameter between the three-dimensional coordinate system of a designated image acquisition device in the image acquisition system and the world coordinate system of the target object, and a second extrinsic parameter between the three-dimensional coordinate systems of the designated image acquisition device and other image acquisition devices, a first mapping relationship is established between the three-dimensional coordinates of the feature point in the world coordinate system of the target object and its three-dimensional coordinates in the three-dimensional coordinate system of each image acquisition device, including:

[0063] The first mapping relationship is established using the following formula:

[0064]

[0065] in, This represents the three-dimensional coordinates of the feature point in the three-dimensional coordinate system of the i-th image acquisition device when the target object is in the t-th pose, t = 0, 1, ..., n, i = 0, 1, ..., k, ..., m-1, and c represents the image acquisition device. This indicates that the first extrinsic parameter changes with the pose of the target object. i T k The second extrinsic parameter relates to the relative positions of multiple image acquisition devices, where k indicates that the k-th image acquisition device is the specified image acquisition device, and current_shape_3D. t This represents the three-dimensional coordinates of the feature point in the world coordinate system when it is in the t-th pose.

[0066] In one embodiment, the designated image acquisition device may include: the image acquisition device most directly facing the target face among the plurality of image acquisition devices; or an image acquisition device whose two-dimensional coordinate plane is parallel to the plane formed by the lines connecting the eyes and mouth of the target face.

[0067] In this embodiment, the specified image acquisition device can be an image acquisition device that can capture as many feature points as possible spread out, such as an image acquisition device facing the target face, or an image acquisition device whose two-dimensional coordinate plane is parallel to the plane connecting the eyes and mouth of the target face.

[0068] By selecting a suitable image acquisition device as the designated image acquisition device, the number of overlapping feature points or feature points in close proximity on the image can be reduced, avoiding a decrease in information during subsequent error calculation, which is beneficial to improving the accuracy of external parameter estimation.

[0069] For example, refer to Figure 4 The distance between the left edge point (feature point 1) and the nose (feature points 28-31) can be used to determine whether the target face is turning left or right. Similarly, the distance between the eyebrow and mouth and the mouth and chin can be used to determine whether the target face is tilted up or down, thus determining whether the target face being viewed by the image acquisition device is facing forward.

[0070] For example, the distance between the image acquisition device and the target face's eyes and mouth can be used to determine whether the two-dimensional coordinate plane of the image acquisition device is parallel to the plane formed by the lines connecting the target face's eyes and mouth.

[0071] In one embodiment, it may further include: determining the three-dimensional coordinates of the feature points of the target face in the world coordinate system of the target face using the following formula:

[0072] current_shape_3D t =mean_shape + pv·params t

[0073] Among them, current_shape_3D t This represents the three-dimensional coordinates of the feature point in the world coordinate system of the target face when the target object is in the t-th pose. `mean_shape` represents the preset average face model, and `pv` represents the face feature vector. Both `mean_shape` and `pv` are fixed constants. `params` t The iteration parameters represent the parameters at the t-th pose of the target object, which change as the coordinates of the feature points change. For the method of determining the three-dimensional coordinates of the feature points in the world coordinate system of the target face in this embodiment, please refer to patent document with application number 202280004769.3.

[0074] In other embodiments, the three-dimensional coordinates of feature points in the world coordinate system of the target face can also be determined by using a depth image acquisition device. Those skilled in the art can adopt appropriate methods as needed.

[0075] In one embodiment, based on the first mapping relationship, establishing a second mapping relationship between the three-dimensional coordinates of the feature point in the three-dimensional coordinate system of each image acquisition device and its second two-dimensional coordinates in the two-dimensional coordinate system of the image acquisition device includes:

[0076] The second mapping relationship is established using the following formula:

[0077]

[0078] in, This represents the second two-dimensional coordinate of the j-th feature point in the two-dimensional coordinate system of the i-th image acquisition device when the target object is in the t-th pose, t = 0, 1, ..., n, i = 0, 1, ..., m-1, j = 0, 1, ..., 67, c represents the image acquisition device, K i This represents the intrinsic parameters of the i-th image acquisition device. This represents the three-dimensional coordinates of the j-th feature point in the three-dimensional coordinate system of the i-th image acquisition device when the target object is in the t-th pose.

[0079] Wherein, when the target object is in the t-th pose, the three-dimensional coordinates of the feature point in the world coordinate system can be represented as:

[0080] In one embodiment, for multiple feature points in multiple images of a target object in different poses, based on the second mapping relationship, the error between the second two-dimensional coordinates and the corresponding first two-dimensional coordinates is determined, including:

[0081] The error is determined using the following formula:

[0082]

[0083] Wherein, error_All represents the error between the second two-dimensional coordinate and the corresponding first two-dimensional coordinate. This represents the second two-dimensional coordinate of the j-th feature point in the two-dimensional coordinate system of the i-th image acquisition device when the target object is in the t-th pose, t = 0, 1, ..., n, i = 0, 1, ..., m-1, j = 0, 1, ..., 67, c represents the image acquisition device, Landmarks_2d t,i,j This represents the first two-dimensional coordinates of the j-th feature point in the image acquired by the i-th image acquisition device when the target object is in the t-th pose.

[0084] Wherein, when the target object is in the t-th pose, the error between the second two-dimensional coordinates and the first two-dimensional coordinates of multiple feature points corresponding to the i-th image acquisition device can be expressed by the following formula:

[0085]

[0086] For n poses of the target object, the error between the second two-dimensional coordinates and the first two-dimensional coordinates of multiple feature points corresponding to the i-th image acquisition device can be expressed by the following formula:

[0087]

[0088] For n poses of the target object, the error between the second two-dimensional coordinates and the first two-dimensional coordinates of multiple feature points corresponding to m-1 image acquisition devices can be expressed by the following formula:

[0089]

[0090] That is, the error between the first two-dimensional coordinates and the second two-dimensional coordinates mentioned above.

[0091] In this embodiment, the distance difference between the first two-dimensional coordinates and the second two-dimensional coordinates of multiple feature points in multiple images of the target object under different poses is used to represent the error between the first two-dimensional coordinates and the second two-dimensional coordinates. Thus, when the error meets the preset error condition, the first external parameter and the second external parameter can be determined.

[0092] In one embodiment, the method may include determining the error that satisfies a preset error condition using the least squares method, and then determining the first extrinsic parameter and the second extrinsic parameter. For example, the error that satisfies the preset error condition can be found using a ceres-solver. In other embodiments, the error that satisfies the preset error condition can also be determined using methods such as gradient descent or Newton's method. The preset error condition may include, for example, an error not exceeding 5, 10, etc., and may also be set as a percentage if possible; this application does not specifically limit this.

[0093] By determining the error that satisfies the preset error conditions, the first external parameter can be solved. Second external parameter i T k and iteration parameters params t .

[0094] In one embodiment, it may further include: performing distortion correction processing on each image to eliminate image distortion caused by each image acquisition device itself.

[0095] Typically, the lens of an image acquisition device is not perfectly parallel to the image screen, resulting in tangential distortion; radial distortion occurs due to the bending of light rays. Geometric correction of the image can be achieved using coordinate transformations to correct these distortions.

[0096] In one implementation,

[0097] Radial distortion of the image is corrected using the following formula:

[0098] x0=x(1+k1r 2 +k2r 4 +k3r 6 )

[0099] y0=y(1+k1r 2 +k2r 4 +k3r 6 )

[0100] Where (x0, y0) are the coordinates of any point on the image before correction, (x, y) are the coordinates of any point on the image after correction, and r 2 =x 2 +y 2 , indicating that the farther away from the center point on the image, the greater the distortion, and k1~k3 represent the radial distortion coefficients;

[0101] The tangential distortion of the image is corrected using the following formula:

[0102] x0=x+[2p1xy+p2(r 2 +2x 2 )]

[0103] y0=x+[2p2xy+p1(r 2 +2y 2 )]

[0104] Where (x0, y0) are the coordinates of any point on the image before correction, (x, y) are the coordinates of any point on the image after correction, and p1 and p2 represent the tangential distortion coefficients.

[0105] By performing the coordinate transformation described above on each point in the image, the corrected image can be obtained. The radial distortion coefficients k1 to k3 and the tangential distortion coefficients p1 and p2 can be obtained using software algorithms, such as the CameraCalibrator toolbox in MATLAB or the calibrate Camera function in the OpenCV library. This application does not impose specific limitations on these methods.

[0106] In one embodiment, after performing distortion correction on each image, the process may further include updating the intrinsic parameters of each image acquisition device. The intrinsic parameters of the image acquisition device can be updated based on the distortion-corrected pixel coordinates.

[0107] Using the method of this invention, images containing the user's face can be captured without the user's awareness. The external parameters of the image acquisition device can be calibrated using the user's face image. The entire process does not require professional operation or guidance, nor does it require the user's active cooperation. It is simple to operate and the user can complete the calibration by himself. Compared with the calibration methods in related technologies, it can bring a better user experience.

[0108] This embodiment provides a computing device, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the steps of the above-described external parameter calibration method for an image acquisition device.

[0109] In one embodiment, the computing device may include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0110] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash FLASH RAM). Memory is an example of computer-readable media.

[0111] This embodiment provides an image acquisition system, including multiple image acquisition devices and the aforementioned computing device.

[0112] The image acquisition system of this embodiment can calibrate the external parameters of each image acquisition device in the image acquisition system using the user's facial image without the user's awareness. The whole process does not require the user's active cooperation or operation by professional technicians and can be completed by the user himself, which can bring a better user experience.

[0113] This embodiment provides a storage medium storing a computer program. When the computer program is executed by the processor, it implements the steps of the above-described external parameter calibration method for the image acquisition device.

[0114] Computer programs can use any combination of one or more storage media. The storage media can be a readable signal medium or a readable storage medium.

[0115] Readable storage media may include, for example, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media may include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0116] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, such as electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any storage medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0117] The computer program contained on the storage medium can be transmitted using any suitable medium, such as wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0118] Computer programs for performing the operations of this invention can be written in any combination of one or more programming languages. Programming languages ​​may include object-oriented programming languages—such as Java, C++, etc.—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The computer program may execute entirely on the user's computing device, partially on the user's device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device may be connected to the user's computing device via any type of network (e.g., including a local area network or a wide area network), or it may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0119] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application. When the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0120] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate.

[0121] It should be understood that the exemplary embodiments described herein can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. These embodiments are provided so that the disclosure of this application is thorough and complete, and that the concept of these exemplary embodiments is fully conveyed to those skilled in the art, and should not be construed as limiting the invention.

[0122] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for calibrating the external parameters of an image acquisition device, characterized in that, include: Acquire multiple images of a target object from multiple different angles, wherein the multiple images are acquired by multiple image acquisition devices of an image acquisition system, and the target object includes a target human face; Extract multiple feature points of the target object and determine the first two-dimensional coordinates of each feature point in each image; Based on the first external parameter between the three-dimensional coordinate system of the specified image acquisition device in the image acquisition system and the world coordinate system of the target object, and the second external parameter between the three-dimensional coordinate systems of the specified image acquisition device and other image acquisition devices, a first mapping relationship is established between the three-dimensional coordinates of the feature point in the world coordinate system of the target object and its three-dimensional coordinates in the three-dimensional coordinate system of each image acquisition device. Based on the first mapping relationship, a second mapping relationship is established between the three-dimensional coordinates of the feature point in the three-dimensional coordinate system of each image acquisition device and its second two-dimensional coordinates in the two-dimensional coordinate system of the image acquisition device; For multiple feature points in multiple images, based on the second mapping relationship, the error between the second two-dimensional coordinates and the corresponding first two-dimensional coordinates is determined. When the error meets the preset error condition, the first extrinsic parameter and the second extrinsic parameter are determined.

2. The external parameter calibration method for the image acquisition device according to claim 1, characterized in that, Acquire multiple images of the target object from different angles, including: Acquire multiple images of the target object from different angles when it is in different poses; For multiple feature points in multiple images, based on the second mapping relationship, the error between the second two-dimensional coordinates and the corresponding first two-dimensional coordinates is determined. When the error meets a preset error condition, the first extrinsic parameter and the second extrinsic parameter are determined, including: For multiple feature points in multiple images of a target object in different poses, based on the second mapping relationship, the error between the second two-dimensional coordinates and the corresponding first two-dimensional coordinates is determined. When the error meets the preset error condition, the first extrinsic parameter and the second extrinsic parameter are determined.

3. The external parameter calibration method for the image acquisition device according to claim 1, characterized in that, Extract multiple feature points of the target object and determine the first two-dimensional coordinates of each feature point in each image, including: Using OpenCV algorithms or machine learning models, multiple feature points of the target face are extracted, and the first two-dimensional coordinates of each feature point in each image are determined.

4. The external parameter calibration method for the image acquisition device according to claim 2, characterized in that, Based on a first external parameter between the three-dimensional coordinate system of a designated image acquisition device in the image acquisition system and the world coordinate system of the target object, and a second external parameter between the three-dimensional coordinate systems of the designated image acquisition device and other image acquisition devices, a first mapping relationship is established between the three-dimensional coordinates of the feature point in the world coordinate system of the target object and its three-dimensional coordinates in the three-dimensional coordinate system of each image acquisition device, including: The first mapping relationship is established using the following formula: in, This represents the three-dimensional coordinates of the feature point in the three-dimensional coordinate system of the i-th image acquisition device when the target object is in the t-th pose, t = 0, 1, ..., n, i = 0, 1, ..., k, ..., m-1, and c represents the image acquisition device. This represents the first extrinsic parameter. i T k The second extrinsic parameter is represented by k, which indicates that the k-th image acquisition device is a specified image acquisition device, and current_shape_3D. t This represents the three-dimensional coordinates of the feature point in the world coordinate system when it is in the t-th pose.

5. The external parameter calibration method for the image acquisition device according to claim 2, characterized in that, Based on the first mapping relationship, a second mapping relationship is established between the three-dimensional coordinates of the feature point in the three-dimensional coordinate system of each image acquisition device and its second two-dimensional coordinates in the two-dimensional coordinate system of the image acquisition device, including: The second mapping relationship is established using the following formula: in, This represents the second two-dimensional coordinate of the j-th feature point in the two-dimensional coordinate system of the i-th image acquisition device when the target object is in the t-th pose, t = 0, 1, ..., n, i = 0, 1, ..., m-1, j = 0, 1, ..., 67, c represents the image acquisition device, K i This represents the intrinsic parameters of the i-th image acquisition device. This represents the three-dimensional coordinates of the j-th feature point in the three-dimensional coordinate system of the i-th image acquisition device when the target object is in the t-th pose.

6. The external parameter calibration method for the image acquisition device according to claim 2, characterized in that, For multiple feature points in multiple images of a target object in different poses, based on the second mapping relationship, the error between the second two-dimensional coordinates and the corresponding first two-dimensional coordinates is determined, including: The error is determined using the following formula: Wherein, error_All represents the error between the second two-dimensional coordinate and the corresponding first two-dimensional coordinate. This represents the second two-dimensional coordinate of the j-th feature point in the two-dimensional coordinate system of the i-th image acquisition device when the target object is in the t-th pose, t = 0, 1, ..., n, i = 0, 1, ..., m-1, j = 0, 1, ..., 67, c represents the image acquisition device, Landmarks_2d t,i,j This represents the first two-dimensional coordinates of the j-th feature point in the image acquired by the i-th image acquisition device when the target object is in the t-th pose.

7. The external parameter calibration method for the image acquisition device according to claim 1, characterized in that, The designated image acquisition device includes: The image acquisition device that is directly facing the target face among the plurality of image acquisition devices; or An image acquisition device whose two-dimensional coordinate plane is parallel to the plane formed by the lines connecting the eyes and mouth of the target face.

8. The external parameter calibration method for the image acquisition device according to claim 1, characterized in that, include: The least squares method is used to determine the error that meets the preset error conditions, and then the first extrinsic parameter and the second extrinsic parameter are determined.

9. The external parameter calibration method for the image acquisition device according to claim 1, characterized in that, Also includes: Each image undergoes distortion correction processing to eliminate image distortion caused by each image acquisition device itself.

10. The external parameter calibration method for the image acquisition device according to claim 9, characterized in that, The distortion includes radial distortion and tangential distortion; Radial distortion of the image is corrected using the following formula: x0=x(1+k1r 2 +k2r 4 +k3r 6 ) y0=y(1+k1r 2 +k2r 4 +k3r 6 ) Where (x0, y0) are the coordinates of any point on the image before correction, (x, y) are the coordinates of any point on the image after correction, and r 2 =x 2 +y 2 k1~k3 represent the radial distortion coefficients; The tangential distortion of the image is corrected using the following formula: x0=x+[2p1xy+p2(r 2 +2x 2 )] y0=x+[2p2xy+p1(r 2 +2y 2 )] Where (x0,y0) are the coordinates of any point on the image before correction, (x,y) are the coordinates of any point on the image after correction, and p1 and p2 represent the tangential distortion coefficients.

11. The external parameter calibration method for the image acquisition device according to claim 1, characterized in that, Also includes: The three-dimensional coordinates of the feature point in the world coordinate system of the target face are determined using the following formula: current_shape_3D t =mean_shape+pv·params t Among them, current_shape_3D t This represents the three-dimensional coordinates of the feature point in the world coordinate system of the target face when the target object is in the t-th pose. `mean_shape` represents the preset average face model, `pv` represents the face feature vector, and both `mean_shape` and `pv` are fixed constants. `params` t This represents the iteration parameters when the target object is in the t-th pose.

12. A computing device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the steps of the external parameter calibration method for the image acquisition device as described in any one of claims 1 to 11.

13. An image acquisition system, characterized in that, It includes multiple image acquisition devices and the computing device as described in claim 12.

14. A storage medium storing a computer program that, when executed by a processor, implements the steps of the external parameter calibration method for an image acquisition device as described in any one of claims 1 to 11.

Images