A virtual reality device, camera calibration apparatus and method

Abstract

This application provides a virtual reality device, a camera calibration apparatus, and a method. Initial values ​​of inter-calibration extrinsic parameters are obtained based on the positions of the master and slave calibration boards. These inter-calibration extrinsic parameters characterize the positional relationship between the master and slave calibration boards. A camera is then controlled to capture a calibration image of the slave calibration board. Calibration corner points in the calibration image are identified, and their corresponding pixel coordinates are obtained. The world coordinates of the corresponding calibration corner points in the slave calibration board are also obtained. A parameter optimization flag is detected. If the flag indicates optimization is to be performed, the inter-calibration extrinsic parameters are optimized based on the pixel coordinates, world coordinates, and the initial values ​​of the inter-calibration extrinsic parameters to complete camera calibration. This solves the problem of unknown or inaccurate inter-calibration extrinsic parameters in scenarios with multiple calibration boards and defined inter-calibration extrinsic parameters, such as when adjusting the camera arrangement or the placement of the calibration boards.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to a virtual reality device, camera calibration apparatus and method. Background Technology

[0002] Virtual Reality (VR) technology is a display technology that simulates virtual environments using computers, thereby creating a sense of immersion. A VR device is a device that uses VR technology to present virtual images to users. Typically, a VR device includes an imaging system composed of multiple cameras. In the process of presenting virtual images, the VR device uses machine vision / computer vision technology to perceive, recognize, and understand the three-dimensional scene of the objective world. Based on this, it reconstructs the visible parts of three-dimensional objects from two-dimensional images using the imaging system.

[0003] The basic principle of visual measurement is to acquire information about an object using an image sensor and establish a mathematical model between the object's coordinates and the image coordinates. By solving for the parameters in the model, the mapping relationship between the object's coordinates and the image coordinates is determined. The object's coordinates are then obtained from the image coordinates, and thus the actual size of the object is determined. The process of solving the model is the same as the camera calibration process; therefore, the calibration accuracy directly affects the measurement accuracy.

[0004] Currently, in camera calibration, the detection coverage area and calibration accuracy are improved by adjusting the placement of calibration plates, i.e., the inter-calibration plate extrinsic parameters. Typically, these extrinsic parameters can be obtained through positional measurements after the calibration plates are fixed in place. Alternatively, the calibration plate positions can be designed and the calibration plates installed based on the determined extrinsic parameters. Therefore, in scenarios with multiple calibration plates and defined inter-calibration plate extrinsic parameters, adjusting the camera arrangement or the placement of the calibration plates can lead to unknown or inaccurate extrinsic parameters. This affects calibration accuracy and reduces the user experience when using virtual reality devices. Summary of the Invention

[0005] This application provides a virtual reality device, a camera calibration apparatus, and a method to solve the problem that the extrinsic parameters between calibration plates may be unknown or inaccurate when adjusting the arrangement of cameras or the placement of calibration plates in scenarios where there are multiple calibration plates and definite extrinsic parameters between the calibration plates.

[0006] In a first aspect, this application provides a virtual reality device, comprising:

[0007] The monitor is configured to display the user interface;

[0008] The camera is used to acquire calibration images corresponding to the calibration board;

[0009] The controller is configured as follows:

[0010] The initial values ​​of the inter-plate extrinsic parameters are obtained based on the positions of the master calibration plate and the slave calibration plate. The inter-plate extrinsic parameters are used to characterize the positional relationship between the master calibration plate and the slave calibration plate.

[0011] Control the camera to capture the calibration image from the calibration board;

[0012] Identify the calibration corner points in the calibration image and obtain the pixel coordinates of the calibration corner points in the calibration image;

[0013] Obtain the world coordinates of the calibration corner point in the calibration plate;

[0014] If the parameter optimization flag is set to "perform optimization", then the inter-board extrinsic parameters are optimized based on the pixel coordinates, the world coordinates, and the initial values ​​of the inter-board extrinsic parameters, so as to complete the camera calibration based on the optimized inter-board extrinsic parameters.

[0015] Secondly, this application also provides a camera calibration device, comprising:

[0016] The initial value acquisition module for inter-board extrinsic parameters is used to acquire the initial values ​​of inter-board extrinsic parameters based on the positions of the master calibration board and the slave calibration board. The inter-board extrinsic parameters are used to characterize the positional relationship between the master calibration board and the slave calibration board.

[0017] A calibration image acquisition module is used to acquire calibration images of the calibration board captured by the camera;

[0018] The calibration corner point determination module is used to identify calibration corner points in the calibration image, obtain the pixel coordinates of the calibration corner points in the calibration image, and obtain the world coordinates of the calibration corner points in the calibration board according to the identifiers corresponding to the calibration corner points.

[0019] The inter-board extrinsic parameter optimization module is used to detect parameter optimization flags. If the parameter optimization flag indicates that optimization is to be performed, the inter-board extrinsic parameters are optimized based on the pixel coordinates, the world coordinates, and the initial values ​​of the inter-board extrinsic parameters, so as to complete the camera calibration based on the optimized inter-board extrinsic parameters.

[0020] Thirdly, this application also provides a camera calibration method, which specifically includes the following steps:

[0021] The initial values ​​of the inter-plate extrinsic parameters are obtained based on the positions of the master calibration plate and the slave calibration plate. The inter-plate extrinsic parameters are used to characterize the positional relationship between the master calibration plate and the slave calibration plate.

[0022] Control the camera to capture the calibration image from the calibration board;

[0023] Identify the calibration corner points in the calibration image and obtain the pixel coordinates of the calibration corner points in the calibration image;

[0024] Obtain the world coordinates of the calibration corner point in the calibration plate;

[0025] If the parameter optimization flag is set to "perform optimization", then the inter-board extrinsic parameters are optimized based on the pixel coordinates, the world coordinates, and the initial values ​​of the inter-board extrinsic parameters, so as to complete the camera calibration based on the optimized inter-board extrinsic parameters.

[0026] As can be seen from the above technical solution, this application provides a virtual reality device, a camera calibration apparatus, and a method. Initial values ​​of inter-calibration extrinsic parameters are obtained based on the positions of the main calibration board and the slave calibration board, whereby the inter-calibration extrinsic parameters characterize the positional relationship between the main calibration board and the slave calibration board. The camera is controlled to capture a calibration image of the slave calibration board. Calibration corner points in the calibration image are identified, and the pixel coordinates corresponding to the calibration corner points in the calibration image are obtained. The world coordinates corresponding to the calibration corner points in the slave calibration board are obtained. A parameter optimization flag is detected; if the parameter optimization flag indicates that optimization is to be performed, the inter-calibration extrinsic parameters are optimized based on the pixel coordinates, the world coordinates, and the initial values ​​of the inter-calibration extrinsic parameters, to complete camera calibration based on the optimized inter-calibration extrinsic parameters. This solves the problem of unknown or inaccurate inter-calibration extrinsic parameters in scenarios with multiple calibration boards and definite inter-calibration extrinsic parameters, such as when adjusting the camera arrangement or the placement of the calibration boards. Attached Figure Description

[0027] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of a display system structure including a virtual reality device according to some embodiments of this application;

[0029] Figure 2 This is a schematic diagram of the global interface of a VR scene according to some embodiments of this application;

[0030] Figure 3 This is a schematic diagram illustrating the transformation of a three-dimensional object into a two-dimensional image according to some embodiments of this application;

[0031] Figure 4 This is a schematic diagram illustrating the coordinate system transformation relationships according to some embodiments of this application;

[0032] Figure 5This is a schematic diagram illustrating distortion according to some embodiments of this application;

[0033] Figure 6 This is a schematic diagram illustrating camera calibration of a virtual reality device 500 according to some embodiments of this application;

[0034] Figure 7 This is a schematic diagram illustrating a camera imaging model according to some embodiments of this application;

[0035] Figure 8 This is an interactive diagram illustrating camera calibration based on parameter optimization identifiers according to some embodiments of this application;

[0036] Figure 9 This is a flowchart illustrating camera calibration in a virtual reality device 500 according to some embodiments of this application;

[0037] Figure 10 This is a schematic diagram illustrating the processing parameters and corresponding error thresholds according to some embodiments of this application;

[0038] Figure 11 This is a schematic diagram of the camera calibration device according to some embodiments of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the exemplary embodiments of this application clearer, the technical solutions in the exemplary embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0040] Based on the exemplary embodiments shown in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete technical solution on its own.

[0041] It should be understood that the terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate, for example, to allow implementation in orders other than those given in the embodiments illustrated or described in this application.

[0042] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0043] As used in this application, the term "module" means any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0044] Throughout this specification, references to "multiple embodiments," "some embodiments," "one embodiment," or simply "embodiment" indicate that a specific feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. Therefore, phrases such as "in multiple embodiments," "in some embodiments," "in at least another embodiment," or "in an embodiment" appearing throughout this specification do not necessarily refer to the same embodiment. Furthermore, in one or more embodiments, specific features, structures, or characteristics can be combined in any suitable manner. Therefore, without limitation, a specific feature, structure, or characteristic shown or described in connection with one embodiment may be combined, in whole or in part, with features, structures, or characteristics of one or more other embodiments. Such modifications and variations are intended to be included within the scope of this application.

[0045] In this application embodiment, the virtual reality device 500 generally refers to a display device that can be worn on a user's face to provide an immersive experience, including but not limited to VR glasses, augmented reality (AR) devices, VR gaming devices, mobile computing devices, and other wearable computers. Some embodiments of this application use VR glasses as an example to illustrate the technical solution; it should be understood that the provided technical solution can also be applied to other types of virtual reality devices. The virtual reality device 500 can operate independently or be connected as an external device to other smart display devices, wherein the display device can be a smart TV, computer, tablet computer, server, etc.

[0046] The virtual reality device 500, when worn on a user's face, displays media assets, providing close-up images to the user's eyes for an immersive experience. To present the media assets, the virtual reality device 500 can include multiple components for displaying the images and for wearing on the face. Taking VR glasses as an example, the virtual reality device 500 can include a shell, positioning fixtures, an optical system, a display component, a posture detection circuit, and interface circuits. In practical applications, the optical system, display component, posture detection circuit, and interface circuit can be housed within the shell to display the specific images; positioning fixtures are connected to both sides of the shell for wearing on the user's face.

[0047] When in use, the attitude detection circuit has built-in attitude detection components such as gravity acceleration sensor and gyroscope. When the user's head moves or turns, the user's attitude can be detected and the detected attitude data can be transmitted to the controller and other processing components, so that the processing components can adjust the specific screen content in the display component according to the detected attitude data.

[0048] like Figure 1 As shown, in some embodiments, the virtual reality device 500 can be connected to the display device 200 and a network-based display system can be built between it and the server 400. Data interaction can be performed in real time between the virtual reality device 500, the display device 200 and the server 400. For example, the display device 200 can obtain media data from the server 400 and play it, and transmit specific screen content to the virtual reality device 500 for display.

[0049] The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. The specific type, size, and resolution of the display device are not limited. Those skilled in the art will understand that the display device 200 can be modified in terms of performance and configuration as needed. The display device 200 can provide broadcast television reception functionality and may also include, but is not limited to, intelligent network television functionality with computer support, including but not limited to, internet television, smart television, and Internet Protocol television (IPTV).

[0050] Display device 200 and virtual reality device 500 also communicate with server 400 via various communication methods. Display device 200 and virtual reality device 500 can communicate via local area network (LAN), wireless local area network (WLAN), and other networks. Server 400 can provide display device 200 with various content and interactive features. For example, display device 200 can interact by sending and receiving information, as well as electronic program guides (EPGs), receiving software updates, or accessing remotely stored digital media libraries. Server 400 can be a cluster or multiple clusters, and may include one or more types of servers. Other network services such as video-on-demand and advertising services can be provided through server 400.

[0051] During data interaction, users can operate the display device 200 via the mobile terminal 300 and the remote control 100. The mobile terminal 300 and the remote control 100 can communicate with the display device 200 via a direct wireless connection or a non-direct connection. Specifically, in some embodiments, the mobile terminal 300 and the remote control 100 can communicate with the display device 200 via direct connection methods such as Bluetooth or infrared. When sending control commands, the mobile terminal 300 and the remote control 100 can directly transmit the control command data to the display device 200 via Bluetooth or infrared.

[0052] In other embodiments, the mobile terminal 300 and the remote controller 100 can also access the same wireless network as the display device 200 via a wireless router to establish a non-direct connection communication with the display device 200 through the wireless network. When sending control commands, the mobile terminal 300 and the remote controller 100 can first send the control command data to the wireless router, and then the wireless router forwards the control command data to the display device 200.

[0053] In some embodiments, users can also use the mobile terminal 300 and the remote control 100 to interact directly with the virtual reality device 500. For example, the mobile terminal 300 and the remote control 100 can be used as controllers in a virtual reality scene to achieve functions such as motion-sensing interaction.

[0054] In some embodiments, the display component of the virtual reality device 500 includes a display screen and driving circuitry associated with the display screen. To present a concrete image and provide a stereoscopic effect, the display component may include two display screens, corresponding to the user's left and right eyes respectively. When presenting a 3D effect, the content displayed on the left and right screens will be slightly different, and can respectively display the left and right cameras used during the filming of the 3D source material. Because the user observes the image content with their left and right eyes, a more stereoscopic display image can be observed when the device is worn.

[0055] The optical system in the virtual reality device 500 is an optical module composed of multiple lenses. Positioned between the user's eyes and the display screen, the optical system increases the optical path through the refraction of light signals by the lenses and the polarization effect of polarizers on the lenses, ensuring that the content displayed is clearly presented within the user's field of vision. Furthermore, to accommodate different users' visual acuity, the optical system also supports focusing. This involves adjusting the position of one or more lenses using a focusing component, changing the distance between the lenses, and thus altering the optical path and adjusting the image clarity.

[0056] The interface circuit of the virtual reality device 500 can be used to transmit interactive data. Besides transmitting posture data and display content data, in practical applications, the virtual reality device 500 can also connect to other display devices or peripherals through the interface circuit to achieve more complex functions through data interaction with the connected devices. For example, the virtual reality device 500 can connect to a display device through the interface circuit to output the displayed image to the display device in real time. As another example, the virtual reality device 500 can also connect to a controller through the interface circuit, which can be held and operated by the user to perform related operations in the VR user interface.

[0057] The VR user interface can be presented in various different UI layouts based on user operations. For example, the user interface may include a global interface, such as the global UI after the AR / VR terminal is started. Figure 2 As shown, the global UI can be displayed on the display screen of the AR / VR terminal or on the display device's monitor. The global UI may include a recommended content area 1, a business category extension area 2, an application quick operation entry area 3, and a floating element area 4.

[0058] When presenting highly stereoscopic images, the virtual reality device 500 typically utilizes machine vision / computer vision technology. In the fields of machine vision / computer vision, the camera in the virtual reality device 500 is usually used to image objects and scenes in the objective world, enabling 3D reconstruction of these objects and scenes based on the camera's images. This allows for machine-based 3D recognition of the objective world, such as identifying the shape, size, distance from the viewpoint, texture, and motion characteristics of objects. It should be noted that this application only uses the use of machine vision / computer vision technology in the virtual reality device 500 as an example; machine vision / computer vision technology can also be applied to virtual reality, advanced driver assistance systems, unmanned aerial vehicle (UAV) navigation systems, visual inspection, biometric recognition, distance measurement, and 3D reconstruction.

[0059] Furthermore, to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image captured by the camera, the virtual reality device 500 must establish a geometric model of camera imaging. These geometric model parameters can be summarized as camera parameters. In most cases, camera parameters must be obtained through experimentation and calculation; the process of solving for camera parameters through experimentation and calculation is called camera calibration. Camera calibration is a crucial step in fields such as machine vision and computer vision. The accuracy of the calibration results and the stability of the algorithm directly affect the accuracy of image inference results. Therefore, proper camera calibration is a prerequisite for subsequent work, and completing camera calibration quickly and accurately is fundamental to improving the efficiency of subsequent work.

[0060] In some embodiments, the virtual reality device 500 includes an imaging system consisting of multiple cameras. For example, the virtual reality device 500 may have a top dual camera and a bottom dual camera, with the bottom dual camera and the top dual camera positioned and oriented according to a preset angle.

[0061] like Figure 3 As shown, objects in real space are three-dimensional, while their corresponding objects in images are two-dimensional. Therefore, a 3D-to-2D mapping model can be considered to exist between objects in a 3D scene and their corresponding 2D images. This mapping model allows for transformations from 3D to 2D or 2D to 3D between objects in a 3D scene and their corresponding images. It's easy to understand that when a camera captures an image of an object in a 3D scene, the camera can be considered this mapping model, and the camera parameters are the parameters of this mapping model. Therefore, by obtaining the camera parameters, the world coordinates of the object in space can be deduced from the pixel coordinates of the object in the image, thereby enabling functions such as visual detection, biometric recognition, distance measurement, and 3D reconstruction.

[0062] The basic principles and key technical concepts involved in camera calibration will be explained in detail below with reference to accompanying figures and text. It should be noted that some technical concepts may be expressed differently in different ways and according to different conventions. For example, a concept may be expressed as multiple sub-concepts, or several concepts may be summarized into one or more other concepts through splitting or combining. These changes in expression are merely changes in convention and do not constitute a substantial difference from the embodiments of this application.

[0063] Generally, camera calibration involves the following concepts:

[0064] Camera intrinsic parameters (or simply camera intrinsics) refer to parameters related to the camera's inherent characteristics, which can be determined by the positions of optical lenses and photoelectric sensors installed within the camera module. Camera intrinsics may include at least one of the following parameters: focal length, pixel size, optical distortion, white balance, resolution, contrast, vignetting, and shading.

[0065] The camera's external parameters (or simply camera extrinsic parameters) may include at least one of the following:

[0066] World coordinate system: also known as measurement coordinate system, is a three-dimensional Cartesian coordinate system used as a reference to describe the spatial position of the camera and the object being measured in the objective world. The position of the world coordinate system can be freely determined according to the actual situation.

[0067] The camera's external parameters and internal parameters are collectively referred to as camera parameters.

[0068] Camera coordinate system: It is a three-dimensional rectangular coordinate system with the origin located at the optical center of the optical lens (where the optical center is the center point of the lens). The x and y axes are parallel to the two sides of the camera's image plane, and the z axis is the optical axis of the optical lens, which is perpendicular to the imaging plane.

[0069] Pixel coordinates are a two-dimensional Cartesian coordinate system that reflects the arrangement of pixels in a camera's CCD / CMOS chip. The origin of the pixel coordinate system is located at the top left corner of the image, and the u-axis and v-axis are parallel to the two sides of the image plane, respectively. The unit of the coordinate axes in the pixel coordinate system is pixels (integers).

[0070] Image coordinates are a two-dimensional Cartesian coordinate system, with the coordinate axes typically measured in millimeters (mm). The origin is the intersection of the camera's optical axis and the image plane (called the principal point), which is the center of the image. The X and Y axes are parallel to the u and v axes of the pixel coordinate system, respectively. The image coordinate system and the pixel coordinate system are related by translation; that is, they can be obtained through translation.

[0071] Figure 4 This is a schematic diagram illustrating the coordinate system transformation relationships according to some embodiments of this application. The following is in conjunction with... Figure 4 The transformation relationships between various coordinate systems are introduced, and then the camera parameters that need to be calibrated in the embodiments of this application are explained in detail:

[0072] 1. The conversion from the world coordinate system to the camera coordinate system (also known as the camera coordinate system) can be achieved through the following matrix transformations:

[0073]

[0074] Where R is a 3x3 rotation matrix, t is a 3x1 translation vector, (x c y c , z c ,1) T The coordinates in the camera coordinate system are (x) w y w , z w ,1) T These are coordinates in the world coordinate system. Therefore, converting from the world coordinate system to the camera coordinate system actually involves a rotation process and a translation process.

[0075] 2. The conversion from image coordinate system to pixel coordinate system can be achieved through the following matrix transformations:

[0076]

[0077] Where (X, Y, 1)T Let (u, v, 1) be the coordinates in the image coordinate system. T Let dX and dY be the coordinates of the image coordinate system, respectively, and let u0 and v0 be the coordinates of the origin of the image coordinate system in the pixel coordinate system, i.e., the coordinates of the optical center.

[0078] 3. Converting the camera coordinate system to the image coordinate system, as follows: Figure 3 This illustrates a perspective projection process:

[0079] For example, the line connecting any point P in space to the optical center C of the camera is CP, and the intersection point p of CP and the image plane is the projection of the spatial point P onto the image plane. This projection can be achieved through the following matrix transformation:

[0080]

[0081] Where S is the scaling factor (S is not 0), f is the effective focal length of the camera (the distance from the optical center to the image plane), (x c y c , z c ,1) T Let P be the coordinates of point P in the camera coordinate system, (X, Y, 1). T It represents the coordinates of point p in the image coordinate system.

[0082] 4. Convert world coordinate system to pixel coordinate system:

[0083] Based on the transformation matrices between the various coordinate systems shown in 1 to 3 above, the conversion from the world coordinate system to the pixel coordinate system can be obtained through the following matrix transformations:

[0084]

[0085] M1M2X w =MX w ……(4)

[0086] in:

[0087]

[0088] In the above matrices, M is the camera parameter matrix, i.e., the camera parameters; M1 is the camera intrinsic parameter matrix, i.e., the camera's internal parameters; and M2 is the camera extrinsic parameter matrix, i.e., the camera's external parameters. Camera calibration includes solving for the aforementioned intrinsic parameter matrix M1 and / or extrinsic parameter matrix M2. For example, solving for the camera's pixel sizes 1 / dX and 1 / dY, the camera's focal length f, the camera's optical center position (v0, u0), rotation matrix R, and translation variables, etc.

[0089] It should be added that the actual image captured by a camera usually has distortion, which can be called aberration. For example... Figure 5 As shown, distortion can be specifically divided into tangential distortion and radial distortion. Radial distortion originates from the camera lens itself, stemming from the effects of "fisheye" and "barrel" distortion, and is more pronounced at the edges of the image formed by the lens. Tangential distortion is caused by manufacturing defects in the lens or installation deviations that result in the lens itself not being parallel to the image plane.

[0090] During camera calibration, the placement of calibration plates—specifically, the inter-calibration extrinsic parameters—is typically adjusted to improve detection coverage and calibration accuracy. These inter-calibration extrinsic parameters can be obtained through positional measurements after the calibration plates are fixed in place, or by designing and installing the calibration plates based on the defined inter-calibration extrinsic parameters. Therefore, in scenarios with multiple calibration plates and defined inter-calibration extrinsic parameters, adjusting the camera arrangement or the placement of the calibration plates can lead to unknown or inaccurate inter-calibration extrinsic parameters, thus affecting calibration accuracy.

[0091] To avoid issues such as unknown or inaccurate extrinsic parameters between calibration boards, some embodiments provide a virtual reality device, including: a display, a camera, and a controller; wherein the display is configured to display a user interface; the camera is configured to acquire calibration images corresponding to the calibration boards. The controller optimizes the extrinsic parameters between calibration boards to ensure calibration accuracy during camera calibration.

[0092] The following is a detailed description of a virtual reality device and camera calibration method provided in this application.

[0093] Figure 6 This is a schematic diagram illustrating camera calibration of a virtual reality device 500 according to some embodiments of this application. See also Figure 6 This application provides a virtual reality device in which the controller is configured to perform the following steps when executing a focus operation method:

[0094] S1. Obtain the initial values ​​of the inter-plate extrinsic parameters based on the positions of the master calibration plate and the slave calibration plate. The inter-plate extrinsic parameters are used to characterize the positional relationship between the master calibration plate and the slave calibration plate.

[0095] In some embodiments, the technical solution provided in this application is applied to multi-plate calibration. The virtual reality device 500 may include N non-coplanar calibration plates, where N is an integer greater than 2. Non-coplanarity means that the three calibration plates are not parallel to each other, and each pair of plates forms a certain angle with each other. The calibration plates include a master calibration plate and slave calibration plates. For example, the virtual reality device 500 includes one master calibration plate and two slave calibration plates, with the three calibration plates perpendicular to each other. This application does not specifically limit the angle between the calibration plates or the arrangement of the master and slave calibration plates.

[0096] In some embodiments, the controller can obtain the initial values ​​of inter-board extrinsic parameters based on an established initial value mapping table. The initial value mapping table characterizes the relationship between inter-board distances and inter-board angles and the initial values. The inter-board distance is the distance between the master calibration board and the slave calibration boards. Specifically, the inter-board distance refers to the distance between multiple slave calibration boards and the master calibration board, and the inter-board angle refers to the set angle between multiple slave calibration boards and the master calibration board, with the master calibration board as the reference. The corresponding inter-board extrinsic parameters are obtained by measuring the inter-board distances and inter-board angles. An initial value mapping table is established based on the inter-board distances, inter-board angles, and the corresponding inter-board extrinsic parameters.

[0097] It should be noted that the initial value mapping table can also be used to characterize the relationship between the distance between plates and the initial value, or to characterize the relationship between the angle between plates and the initial value.

[0098] For example, in practical applications, multiple inter-board distances and inter-board angles between the primary calibration board and the main calibration board are detected. These inter-board distances and angles are then substituted into an initial value mapping table to find the corresponding inter-board extrinsic parameters. To easily distinguish these from the inter-board extrinsic parameters obtained in subsequent optimizations, the inter-board extrinsic parameters obtained above are referred to as the initial values ​​of the inter-board extrinsic parameters.

[0099] In some embodiments, a preset pattern may be set on the calibration board, such as a checkerboard pattern, a dot pattern, a QR code pattern, or other specific patterns. For the calibration board, any pattern unit, such as any square of a checkerboard pattern or any dot of a dot pattern, can be called a marker. Some feature points, such as calibration corner points, can be determined from the markers for subsequent calculation of camera parameters.

[0100] In some embodiments, each calibration board has a unique corresponding calibration board identifier. The calibration board identifier can be used to distinguish each calibration board, and different calibration boards can be distinguished by the calibration board identifier. For example, the calibration board identifiers corresponding to the three calibration boards are their corresponding IDs: calibration board 01, calibration board 02, and calibration board 03. Thus, the three calibration boards can be distinguished.

[0101] S2. Control the camera to capture a calibration image from the calibration board, identify the calibration corner points in the calibration image, and obtain the pixel coordinates of the corresponding calibration corner points in the calibration image. Also, obtain the world coordinates of the corresponding calibration corner points from the calibration board.

[0102] In some embodiments, the results captured by the camera include photographs and / or videos. The camera can send the captured photographs and videos back to the controller, which stores them in its memory for use in subsequent steps to calculate camera intrinsic parameters, camera extrinsic parameters, and inter-board extrinsic parameters. For example, the camera can send analog signals of photographs and videos to an acquisition device, which performs analog-to-digital conversion on the signals and then sends the resulting digital signals of the photographs and videos to the controller for storage.

[0103] In some embodiments, Figure 7 This is a schematic diagram illustrating a camera imaging model according to some embodiments of this application. See also Figure 7 Before controlling the camera to capture the calibration image of the calibration board, a camera imaging model is established based on the initial values ​​of the inter-board extrinsic parameters. The specific steps for establishing the camera imaging model are as follows: obtain the first transformation relationship between the pixel coordinate system and the image coordinate system, and generate the first transformation matrix according to the first transformation relationship. Obtain the second transformation relationship between the image coordinate system and the camera coordinate system, and generate the second transformation matrix according to the second transformation relationship. Obtain the third transformation relationship between the camera coordinate system and the world coordinate system, and generate the third transformation matrix according to the third transformation relationship. Integrate the first transformation matrix, the second transformation matrix and the third transformation matrix to generate the camera imaging model based on the integrated first transformation matrix, the second transformation matrix and the third transformation matrix. It should be noted that the first transformation matrix, the second transformation matrix and the third transformation matrix correspond to the above transformation matrix for converting the image coordinate system to the pixel coordinate system (formula (2)), the transformation matrix for converting the camera coordinate system to the image coordinate system (formula (3)) and the transformation matrix for converting the world coordinate system to the camera coordinate system (formula (1)). After integrating the three, it is the transformation matrix for converting the world coordinate system to the pixel coordinate system (formula (4)), which constitutes the camera imaging model. This application does not specifically limit the specific transformation relationship and transformation matrix, but only uses the above-mentioned first transformation matrix, second transformation matrix and third transformation matrix as examples.

[0104] For example, when the camera captures a photograph, the controller can detect and obtain calibration corner points from the aforementioned "part or all" of the photograph. Each calibration corner point has pixel coordinates in the photograph and world coordinates. Based on the pixel coordinates and world coordinates of the calibration corner points, the camera intrinsic parameters, camera extrinsic parameters, and camera distortion coefficients in the aforementioned "conversion from world coordinates to pixel coordinates" can be solved.

[0105] For example, when the camera captures a video, the controller can extract at least one video frame from the video, each of which contains part or all of the calibration pattern. The controller can then detect and obtain calibration corner points from these at least one video frame. Each calibration corner point has coordinates in the pixel coordinate system and coordinates in the world coordinate system. Based on the pixel coordinate system coordinates and world coordinate system coordinates of the calibration corner points, the camera intrinsic parameters, camera extrinsic parameters, and camera distortion coefficients in the "world coordinate system to pixel coordinate system conversion" can be solved.

[0106] Therefore, by using the pixel coordinates of the calibrated corner points in the calibration image and the world coordinates of the calibrated corner points in the calibration plate, the camera intrinsic parameters, camera extrinsic parameters, and camera distortion coefficients can be obtained.

[0107] In some embodiments, the controller preprocesses the calibration image captured by the camera before identifying the calibration corner points. For example, the calibration image is converted to grayscale to obtain a grayscale image. The grayscale image is then filtered to obtain a smoothed image. The pixel values ​​of the grayscale image are adjusted based on the smoothed image to obtain candidate images containing gradients and grayscale values. Furthermore, the calibration image is determined based on the candidate images to identify the calibration corner points within the calibration image.

[0108] In some embodiments, the controller can identify calibration corner points in calibration images based on calibration board identifiers, and determine the coordinate information of calibration corner points in other calibration images besides the calibration board itself based on the calibration board identifiers corresponding to the calibration board.

[0109] S3. Detect parameter optimization flag. If the parameter optimization flag is to be optimized, then optimize the inter-board extrinsic parameters based on the pixel coordinates, world coordinates and the initial values ​​of the inter-board extrinsic parameters, so as to complete the camera calibration based on the optimized inter-board extrinsic parameters.

[0110] In some embodiments, Figure 8 This is an interactive diagram illustrating camera calibration based on parameter optimization identifiers according to some embodiments of this application. See also Figure 8 The controller receives a control command indicating parameter optimization, whereby the control command includes a parameter optimization flag. In response to the control command, the parameter optimization flag is detected. If the parameter optimization flag indicates optimization to be performed, the inter-board extrinsic parameters are optimized based on pixel coordinates, world coordinates, and initial values ​​of the inter-board extrinsic parameters, and the optimized inter-board extrinsic parameters are output. The controller then controls the display to show the user interface based on the output optimized inter-board extrinsic parameters.

[0111] The following section will elaborate on the process of optimizing inter-board extrinsic parameters based on pixel coordinates, world coordinates, and initial values ​​of inter-board extrinsic parameters. Figure 9 This is a flowchart illustrating camera calibration for a virtual reality device 500 according to some embodiments of this application. See also... Figure 9 For example, considering distortion and not considering inter-plate extrinsic parameters, the camera imaging model is as follows:

[0112] <u,v> =F d (K,θ d ,θ e | <X w ,Y w Z w >);

[0113] in,<u,v> This represents the pixel coordinates of the calibration corner point on the calibration image. <X w ,Y w Z w > represents the world coordinates of the calibration corner point in the world coordinate system; K represents the camera intrinsic parameters; θ d Represents camera extrinsic parameters; θ e F represents the camera distortion coefficient; d This is a preset function.

[0114] During the calibration process using multiple calibration boards, an optimized set of extrinsic parameters between the boards is constructed by establishing a system of equations connecting the world coordinates and pixel coordinates of the calibration corner points, and then solving for these parameters.

[0115] Considering the inter-plate extrinsic parameters, the camera imaging model is as follows:

[0116] <u,v> =F d (K,θ d ,θ e ,θ b | <X w' ,Y w' Z w' >);

[0117] in, <X w' ,Y w' Z w' >The 3D coordinates of the calibrated corner point in the world coordinate system constructed on its calibration plate; θ b For the inter-board extrinsic parameters of the calibration board; F d This is a preset function;<u,v> The pixel coordinate system, i.e., the 2D coordinates under the calibration image, is constructed for the calibrated corner points on their respective calibration plates.

[0118] Therefore, the camera imaging model considering inter-plate extrinsic parameters will be transformed, and the transformed camera imaging model is as follows:

[0119] θ b ( <X w' ,Y w' Z w' >)= <X w ,Y w Zw >

[0120] Based on the above camera imaging model, the following system of equations is constructed:

[0121]

[0122] Therefore, the optimized inter-plate extrinsic parameters can be obtained using the following formula:

[0123]

[0124] Among them, the inter-plate extrinsic parameters corresponding to the calibration corner points on the main calibration plate are the initial values ​​of the inter-plate extrinsic parameters.

[0125] In some embodiments, the pixel coordinates, world coordinates, and initial values ​​of inter-plate extrinsic parameters are input into the camera imaging model described above. Using Jacobi calculation, parameter optimization, and nonlinear optimization methods, the inter-plate extrinsic parameters, camera intrinsic parameters, camera extrinsic parameters, and camera distortion coefficients corresponding to the calibration corner points on the calibration plate are obtained. The inter-plate extrinsic parameters corresponding to the calibration corner points on the calibration plate can also be referred to as the optimized inter-plate extrinsic parameters.

[0126] It is understood that, based on the process of obtaining the optimized inter-plate extrinsic parameters described above, the embodiments of this application can perform multiple calibrations on the camera, and the optimized inter-plate extrinsic parameters can be used as the initial values ​​of the inter-plate extrinsic parameters at the beginning of the next camera calibration process. This allows the camera intrinsic parameters, camera extrinsic parameters, camera distortion coefficients, and inter-plate extrinsic parameters to be continuously iteratively updated and optimized, saving the solution time of the optimization problem and improving optimization efficiency.

[0127] In some embodiments, if parameter optimization is marked as not performing optimization, pixel coordinates and world coordinates are input into the camera imaging model to obtain the camera's intrinsic parameters, extrinsic parameters, and camera distortion coefficients; and camera calibration is completed based on the intrinsic and extrinsic parameters.

[0128] For example, if the controller detects that the parameter optimization is marked as not performing optimization, it does not establish a camera model and equations that consider inter-board extrinsic parameters. By inputting the pixel coordinates and world coordinates of the calibrated corner points into the camera imaging model, the camera's intrinsic parameters, extrinsic parameters, and camera distortion coefficients can be obtained. This process is the same as S2 above and will not be repeated here.

[0129] In some embodiments, Figure 10 This is a schematic diagram illustrating processing parameters and corresponding error thresholds according to some embodiments of this application. At least one of the internal and external parameters has a corresponding error threshold. If the repeatability of the parameter is greater than the corresponding error threshold, then the calibration pattern from the calibration board is replaced, and / or the number of calibration corner points extracted from the calibration image is adjusted, and / or the image quality from the calibration board is adjusted.

[0130] In some embodiments, different virtual reality devices 500 have different requirements for camera calibration. For example, all parameters in the camera may be parameters to be calibrated, or only some parameters may be calibrated. Thus, when all or some parameters meet their corresponding error thresholds, the obtained parameters can be considered accurate, satisfying the output conditions, and the calibration result can be output. If the error of any parameter does not meet its corresponding error threshold, it is necessary to return to step S1 to continue calibration, iteratively updating to obtain more accurate parameters that meet the calibration requirements. Here, repeatability accuracy refers to the deviation between multiple parameters obtained by repeatedly performing calibration under the same calibration environment. The smaller the deviation, the smaller the parameter error; the larger the deviation, the larger the parameter error.

[0131] For example, the calibration pattern is usually fixed during the calibration process, meaning the number and size of the markers remain unchanged. Therefore, in the next calibration, a new calibration pattern can be obtained by changing the marker pattern and increasing the number of markers, thereby increasing the number of calibration corner points extracted by the controller from the shooting results and improving calibration accuracy.

[0132] For example, the accuracy of corner point extraction can also be improved by adjusting parameters that affect image quality, such as image brightness and contrast.

[0133] The above method optimizes the inter-plate extrinsic parameters based on pixel coordinates, world coordinates, and initial values ​​of the inter-plate extrinsic parameters, and completes camera calibration based on the optimized inter-plate extrinsic parameters. This solves the problem of unknown or inaccurate inter-plate extrinsic parameters in scenarios with multiple calibration plates and definite inter-plate extrinsic parameters, such as when adjusting the camera arrangement or the placement of the calibration plates.

[0134] This application also provides a camera calibration method, which specifically includes the following steps: obtaining initial values ​​of inter-plate extrinsic parameters based on the positions of the master calibration plate and the slave calibration plate, wherein the inter-plate extrinsic parameters are used to characterize the positional relationship between the master calibration plate and the slave calibration plate; controlling the camera to capture a calibration image of the slave calibration plate; identifying calibration corner points in the calibration image and obtaining the pixel coordinates corresponding to the calibration corner points in the calibration image; obtaining the world coordinates corresponding to the calibration corner points in the slave calibration plate; detecting parameter optimization flags; if the parameter optimization flag indicates that optimization is to be performed, optimizing the inter-plate extrinsic parameters based on the pixel coordinates, world coordinates, and the initial values ​​of the inter-plate extrinsic parameters, so as to complete the camera calibration based on the optimized inter-plate extrinsic parameters.

[0135] In some embodiments, the method further includes: in the step of obtaining the initial values ​​of the inter-plate extrinsic parameters, establishing an initial value mapping table, which is used to characterize the correlation between the inter-plate distance and the inter-plate angle and the initial values, wherein the inter-plate distance is the distance between the master calibration plate and the slave calibration plate. The inter-plate distance and the inter-plate angle are detected. Based on the initial value mapping table, the initial values ​​corresponding to the inter-plate distance and the inter-plate angle are obtained.

[0136] In some embodiments, the method further includes: before the step of controlling the camera to capture a calibration image of the calibration board, establishing a camera imaging model based on the initial values ​​of the inter-board extrinsic parameters. Pixel coordinates, world coordinates, and the initial values ​​of the inter-board extrinsic parameters are input into the camera imaging model to obtain optimized inter-board extrinsic parameters.

[0137] In some embodiments, the method further includes: in the step of establishing a camera imaging model, obtaining a first transformation relationship between the pixel coordinate system and the image coordinate system, and generating a first transformation matrix based on the first transformation relationship; obtaining a second transformation relationship between the image coordinate system and the camera coordinate system, and generating a second transformation matrix based on the second transformation relationship; obtaining a third transformation relationship between the camera coordinate system and the world coordinate system, and generating a third transformation matrix based on the third transformation relationship; and integrating the first transformation matrix, the second transformation matrix, and the third transformation matrix to generate a camera imaging model based on the integrated first transformation matrix, the second transformation matrix, and the third transformation matrix.

[0138] In some embodiments, the method further includes: if parameter optimization is marked as not performing optimization, then inputting pixel coordinates and world coordinates into the camera imaging model to obtain the camera's intrinsic parameters, extrinsic parameters, and camera distortion coefficients; to complete camera calibration based on the intrinsic and extrinsic parameters.

[0139] In some embodiments, the method further includes: at least one of the internal parameters and external parameters having a corresponding error threshold; if the repeatability of the parameter is greater than the corresponding error threshold, then the calibration pattern from the calibration board is replaced, and / or the number of calibration corner points extracted from the calibration image is adjusted, and / or the image quality from the calibration board is adjusted.

[0140] In some embodiments, the method further includes: in the step of controlling the camera to capture a calibration image of the calibration board, converting the calibration image to grayscale to obtain a grayscale image; filtering the grayscale image to obtain a smoothed image; adjusting the pixel values ​​of the grayscale image based on the smoothed image to obtain a candidate image containing gradients and grayscale values; and determining a calibration image based on the candidate image to determine calibration corner points in the calibration image.

[0141] See Figure 11This application also provides a camera calibration device, including: an initial value acquisition module for inter-plate extrinsic parameters, used to acquire initial values ​​of inter-plate extrinsic parameters based on the positions of the master calibration plate and the slave calibration plate, wherein the inter-plate extrinsic parameters are used to characterize the positional relationship between the master calibration plate and the slave calibration plate.

[0142] The calibration image acquisition module is used to acquire calibration images of the calibration board taken by the camera.

[0143] The calibration corner point determination module is used to identify calibration corner points in the calibration image and obtain the pixel coordinates of the calibration corner points in the calibration image; and obtain the world coordinates of the calibration corner points in the calibration board according to the identifiers corresponding to the calibration corner points.

[0144] The inter-board extrinsic parameter optimization module is used to detect parameter optimization flags. If the parameter optimization flag indicates that optimization is to be performed, the inter-board extrinsic parameters are optimized based on pixel coordinates, world coordinates, and the initial values ​​of the inter-board extrinsic parameters, so as to complete the camera calibration based on the optimized inter-board extrinsic parameters.

[0145] The same or similar parts among the various embodiments in this specification can be referred to mutually, and will not be repeated here.

[0146] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server 400, or network device, etc.) to execute the methods of various embodiments or certain parts of the embodiments of the present invention.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

[0148] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the exemplary discussion above is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the embodiments and various different variations of embodiments suitable for specific application considerations.

Claims

1. A virtual reality device, characterized in that, include: The monitor is configured to display the user interface; The camera is used to acquire calibration images corresponding to the calibration board; The controller is configured as follows: The initial values ​​of the inter-plate extrinsic parameters are obtained based on the positions of the master calibration plate and the slave calibration plate. The inter-plate extrinsic parameters are used to characterize the positional relationship between the master calibration plate and the slave calibration plate. A camera imaging model is established based on the initial values ​​of the inter-plate extrinsic parameters; Control the camera to capture the calibration image from the calibration board; Identify the calibration corner points in the calibration image and obtain the pixel coordinates of the calibration corner points in the calibration image; Obtain the world coordinates of the calibration corner point in the calibration plate; The pixel coordinates, the world coordinates, and the initial values ​​of the inter-plate extrinsic parameters are input into the camera imaging model to obtain the optimized inter-plate extrinsic parameters. If the parameter optimization flag indicates that optimization is to be performed, then the camera calibration is completed based on the optimized inter-board extrinsic parameters.

2. The virtual reality device according to claim 1, characterized in that, The controller is configured to: In the step of obtaining the initial values ​​of the inter-plate extrinsic parameters, an initial value mapping table is established. The initial value mapping table is used to characterize the relationship between the inter-plate distance and the inter-plate angle and the initial values. The inter-plate distance is the distance between the master calibration plate and the slave calibration plate. Detect the distance between the plates and the angle between the plates; Based on the initial value mapping table, the initial values ​​corresponding to the inter-plate distance and the inter-plate angle are obtained.

3. The virtual reality device according to claim 1, characterized in that, The controller is configured to: In the step of establishing the camera imaging model, a first transformation relationship between the pixel coordinate system and the image coordinate system is obtained, and a first transformation matrix is ​​generated based on the first transformation relationship; Obtain the second transformation relationship between the image coordinate system and the camera coordinate system, and generate a second transformation matrix based on the second transformation relationship; Obtain the third transformation relationship between the camera coordinate system and the world coordinate system, and generate the third transformation matrix based on the third transformation relationship; The first transformation matrix, the second transformation matrix, and the third transformation matrix are integrated to generate the camera imaging model based on the integrated first transformation matrix, the second transformation matrix, and the third transformation matrix.

4. The virtual reality device according to claim 3, characterized in that, The controller is configured to: If the parameter optimization flag is set to not perform optimization, then the pixel coordinates and the world coordinates are input into the camera imaging model to obtain the camera's internal parameters, external parameters, and camera distortion coefficients; and the camera calibration is completed based on the internal parameters and the external parameters.

5. The virtual reality device according to claim 4, characterized in that, At least one of the internal and external parameters has a corresponding error threshold; the controller is configured to: If the repeatability of the parameter is greater than the corresponding error threshold, then the calibration pattern from the calibration board is replaced, and / or the number of calibration corner points extracted from the calibration image is adjusted, and / or the image quality from the calibration board is adjusted.

6. The virtual reality device according to claim 1, characterized in that, The controller is configured to: In the step of controlling the camera to capture the calibration image of the calibration board, the calibration image is processed to grayscale to obtain a grayscale image; The grayscale image is filtered to obtain a smoothed image; The pixel values ​​of the grayscale image are adjusted based on the smoothed image to obtain a candidate image containing gradient and grayscale. The calibration image is determined based on the candidate image, and the calibration corner point is determined in the calibration image.

7. A camera calibration device, characterized in that, include: The initial value acquisition module for inter-board extrinsic parameters is used to acquire the initial values ​​of inter-board extrinsic parameters based on the positions of the master calibration board and the slave calibration board. The inter-board extrinsic parameters are used to characterize the positional relationship between the master calibration board and the slave calibration board. A calibration image acquisition module is used to acquire calibration images of the calibration board captured by the camera; The calibration corner point determination module is used to identify calibration corner points in the calibration image, obtain the pixel coordinates of the calibration corner points in the calibration image, and obtain the world coordinates of the calibration corner points in the calibration board according to the identifiers corresponding to the calibration corner points. The inter-plate extrinsic parameter optimization module is used to establish a camera imaging model based on the initial values ​​of the inter-plate extrinsic parameters; the pixel coordinates, the world coordinates, and the initial values ​​of the inter-plate extrinsic parameters are input into the camera imaging model to obtain the optimized inter-plate extrinsic parameters; If the parameter optimization flag indicates that optimization is to be performed, then the camera calibration is completed based on the optimized inter-board extrinsic parameters.

8. A camera calibration method, characterized in that, The method specifically includes the following steps: The initial values ​​of the inter-plate extrinsic parameters are obtained based on the positions of the master calibration plate and the slave calibration plate. The inter-plate extrinsic parameters are used to characterize the positional relationship between the master calibration plate and the slave calibration plate. A camera imaging model is established based on the initial values ​​of the inter-plate extrinsic parameters; Control the camera to capture the calibration image from the calibration board; Identify the calibration corner points in the calibration image and obtain the pixel coordinates of the calibration corner points in the calibration image; Obtain the world coordinates of the calibration corner point in the calibration plate; The pixel coordinates, the world coordinates, and the initial values ​​of the inter-plate extrinsic parameters are input into the camera imaging model to obtain the optimized inter-plate extrinsic parameters. If the parameter optimization flag indicates that optimization is to be performed, then the camera calibration is completed based on the optimized inter-board extrinsic parameters.

Images