Calibration method, device, equipment and system of light emitting unit and camera

Abstract

The application provides a light emitting unit and camera calibration method, device, equipment and system, wherein the method comprises: acquiring a first image collected by a first camera and / or a second image collected by a second camera in a head-mounted device; determining a plane equation of a plane mirror according to internal and external parameters of the first camera, internal and external parameters of the second camera, the first image and / or the second image; acquiring a third image collected by the first camera and / or a fourth image collected by the second camera; determining a virtual image coordinate of a built-in light emitting unit according to the internal and external parameters of the first camera, the internal and external parameters of the second camera, the third image and / or the fourth image; and calibrating the built-in light emitting unit and the first camera and / or the second camera according to the plane equation of the plane mirror and the virtual image coordinate of the built-in light emitting unit. The application makes the whole calibration process simple and fast, and provides conditions for batch calibration of head-mounted devices provided with light emitting units and cameras.

Description

Technical Field

[0001] This application relates to the field of virtual scene technology, and in particular to a calibration method, apparatus, device and system for a light-emitting unit and a camera. Background Technology

[0002] With the continuous development of Virtual Reality (VR) and Augmented Reality (AR) technologies, eye-tracking technology has been gradually applied to VR / AR devices to enrich their interaction methods. However, human-computer interaction based on eye-tracking technology requires the LED unit on the VR / AR device to emit light towards the user's eyes and control the camera to capture images of the user's eyes. Then, the light spot on the eyeball in the eye image is detected to determine the position of the eyeball, and human-computer interaction is performed based on the eyeball's position. Therefore, pre-calibrating the positional relationship between the LED unit and the camera in the VR / AR device is particularly important.

[0003] Currently, inertial measurement units (IMUs) are commonly used to calibrate the positional relationship between the light-emitting unit and the camera in VR / AR devices. The specific process involves controlling a robotic arm equipped with the VR / AR device to rotate around different coordinate axes to calculate the coordinate transformation relationship between the light-emitting unit and the IMU. Then, the extrinsic parameters between the IMU and the camera are calibrated to indirectly calculate the positional relationship between them. However, this method, besides requiring the IMU as a medium to indirectly calculate the positional relationship, also requires the use of a robotic arm. This is not only costly but also time-consuming due to the long calibration process, inevitably introducing calibration errors between the light-emitting unit and the IMU, and between the IMU and the camera. Summary of the Invention

[0004] This application provides a calibration method, apparatus, device, and system for a light-emitting unit and a camera, making the entire calibration process simple and quick, and providing conditions for the mass calibration of head-mounted devices equipped with light-emitting units and cameras.

[0005] In a first aspect, embodiments of this application provide a calibration method for a light-emitting unit and a camera, including:

[0006] Acquire a first image captured by a first camera and / or a second image captured by a second camera in a head-mounted device, wherein the first image and / or the second image include a real image of an external light-emitting unit and a virtual image of the external light-emitting unit in a plane mirror;

[0007] The plane equation of the plane mirror is determined based on the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the first image, and / or the second image.

[0008] Acquire a third image captured by the first camera and / or a fourth image captured by the second camera, wherein the third image and / or the fourth image includes a virtual image of the built-in light-emitting unit in the head-mounted device in the plane mirror;

[0009] The virtual image coordinates of the built-in light-emitting unit are determined based on the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the third image, and / or the fourth image.

[0010] Based on the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit, the built-in light-emitting unit is calibrated with the first camera and / or the second camera.

[0011] Secondly, embodiments of this application provide a calibration device for a light-emitting unit and a camera, comprising:

[0012] An image acquisition module is used to acquire a first image captured by a first camera and / or a second image captured by a second camera in a head-mounted device. The first image and / or the second image include a real image of an external light-emitting unit and a virtual image of the external light-emitting unit in a plane mirror.

[0013] The equation determination module is used to determine the plane equation of the plane mirror based on the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the first image and / or the second image.

[0014] The image acquisition module is further configured to acquire a third image captured by the first camera and / or a fourth image captured by the second camera, wherein the third image and / or the fourth image includes a virtual image of the built-in light-emitting unit in the head-mounted device in the plane mirror;

[0015] The coordinate determination module is used to determine the virtual image coordinates of the built-in light-emitting unit based on the internal and external parameters of the first camera, the internal and external parameters of the second camera, the third image and / or the fourth image;

[0016] The calibration module is used to calibrate the built-in light-emitting unit and the first camera and / or the second camera based on the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit.

[0017] Thirdly, embodiments of this application provide an electronic device, including:

[0018] A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the calibration method for the light-emitting unit and the camera described in the first aspect embodiment.

[0019] Fourthly, embodiments of this application provide a calibration system for a light-emitting unit and a camera, including: an external light-emitting unit, a plane mirror, a first fixing device, a second fixing device, a head-mounted device, and an electronic device as described in the third aspect;

[0020] The first fixing device is used to fix the external light-emitting unit;

[0021] The second fixing device is used to fix the plane mirror;

[0022] The head-mounted device includes at least a built-in light-emitting unit, a first camera, and a second camera;

[0023] The electronic device is used to perform the calibration method for the light-emitting unit and the camera as described in the first aspect embodiment.

[0024] Fifthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the calibration method for the light-emitting unit and the camera as described in the first aspect embodiment.

[0025] In a sixth aspect, embodiments of this application provide a computer program product containing program instructions that, when executed on an electronic device, cause the electronic device to perform the calibration method for the light-emitting unit and the camera as described in the first aspect embodiment.

[0026] The technical solutions disclosed in the embodiments of this application have at least the following beneficial effects:

[0027] By determining the plane equation of the plane mirror based on the intrinsic and extrinsic parameters of the first and second cameras in the head-mounted device, the first image captured by the first camera, and / or the second image captured by the second camera, and then determining the virtual image coordinates of the built-in light-emitting unit in the head-mounted device based on the intrinsic and extrinsic parameters of the first and second cameras, the third image captured by the first camera, and / or the fourth image captured by the second camera, the virtual image coordinates of the built-in light-emitting unit are determined. Then, based on the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit, the built-in light-emitting unit and the first and / or second cameras are calibrated. This achieves the calibration of the light-emitting unit and camera in the head-mounted device based on the external light-emitting unit and the plane mirror, making the entire calibration process simple and quick, thus providing conditions for the batch calibration of head-mounted devices equipped with light-emitting units and cameras. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a schematic block diagram of a calibration system for a light-emitting unit and a camera provided in an embodiment of this application;

[0030] Figure 2 This is a schematic diagram illustrating the deployment of a head-mounted device, an external light-emitting unit, and a plane mirror, as provided in an embodiment of this application.

[0031] Figure 3 This is a schematic diagram illustrating the deployment of a head-mounted device and a plane mirror according to an embodiment of this application;

[0032] Figure 4 This is a schematic flowchart illustrating a calibration method for a light-emitting unit and a camera provided in an embodiment of this application;

[0033] Figure 5 This is a schematic flowchart illustrating another calibration method for a light-emitting unit and a camera provided in an embodiment of this application;

[0034] Figure 6 This is a schematic flowchart illustrating another calibration method for a light-emitting unit and a camera provided in an embodiment of this application;

[0035] Figure 7 This is a schematic block diagram of a calibration device for a light-emitting unit and a camera provided in an embodiment of this application;

[0036] Figure 8 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0039] This application applies to scenarios involving the calibration of the positional relationship between the light-emitting unit and the camera in virtual reality (VR) or augmented reality (AR) devices. Currently, calibrating the positional relationship between the light-emitting unit and the camera in VR / AR devices using an inertial measurement unit (IMU) requires not only the IMU as a medium to indirectly calculate the positional relationship but also a robotic arm. This is not only costly but also time-consuming due to the long calibration process, inevitably introducing calibration errors related to the light-emitting unit, IMU, and camera. Therefore, this application designs a calibration method that simplifies and speeds up the entire calibration process, providing a basis for the mass calibration of head-mounted devices equipped with light-emitting units and cameras.

[0040] To facilitate understanding of the embodiments of this application, before describing the various embodiments, some concepts involved in all embodiments of this application will be appropriately explained as follows:

[0041] 1) Virtual Reality (VR) is a technology for creating and experiencing virtual worlds. It computationally generates a virtual environment, which is a multi-source information (virtual reality mentioned in this article includes at least visual perception, and may also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.) that realizes the fusion of virtual environment, interactive three-dimensional dynamic visual scenes and simulation of physical behavior, allowing users to immerse themselves in the simulated virtual reality environment, and realize applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0042] 2) Virtual reality devices (VR devices) are terminals that realize virtual reality effects. They can usually be provided in the form of glasses, head-mounted displays (HMDs), or contact lenses to realize visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to these, and they can be further miniaturized or enlarged according to actual needs.

[0043] Optionally, the virtual reality devices described in the embodiments of this application may include, but are not limited to, the following types:

[0044] 2.1) PC-based virtual reality (PCVR) devices utilize a PC for calculations and data output related to virtual reality functions. External PC-based virtual reality devices use the data output from the PC to achieve virtual reality effects.

[0045] 2.2) Mobile virtual reality devices support setting up mobile terminals (such as smartphones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations for virtual reality functions and outputs data to the mobile virtual reality device, such as watching virtual reality videos through the mobile terminal's APP.

[0046] 2.3) All-in-one virtual reality devices have processors for performing virtual functions, thus having independent virtual reality input and output functions. They do not need to be connected to a PC or mobile terminal, and have a high degree of freedom of use.

[0047] 3) AR (Augmented Reality): A technology that calculates the camera's pose parameters in the real world (or 3D world, real world) in real time during image acquisition, and adds virtual elements to the captured images based on these parameters. Virtual elements include, but are not limited to, images, videos, and 3D models. The goal of AR technology is to overlay the virtual world onto the real world on a screen for interactive viewing.

[0048] To clearly illustrate the calibration method of the light-emitting unit and camera provided in this application, the calibration system of the light-emitting unit and camera provided in this application will be described first below.

[0049] Figure 1 This is a schematic block diagram of a calibration system for a light-emitting unit and a camera provided in an embodiment of this application. Figure 1 As shown, the calibration system 1000 for the light-emitting unit and camera provided in this application includes: an external light-emitting unit 110, a plane mirror 120, a first fixing device 130, a second fixing device 140, a head-mounted device 150, and an electronic device 160.

[0050] The first fixing device 130 is used to fix the external light-emitting unit 110;

[0051] The second fixing device 140 is used to fix the plane mirror 120;

[0052] The head-mounted device 150 includes at least a built-in light-emitting unit 151, a first camera 152, and a second camera 153;

[0053] The electronic device 160 is used to perform the calibration method of the light-emitting unit and camera provided in this application, so as to achieve the calibration purpose of the built-in light-emitting unit 151 and the first camera 152 and / or the second camera 153 in the head-mounted device 150.

[0054] In this embodiment, the head-mounted device 150 may be a VR head-mounted device or an AR head-mounted device.

[0055] It should be understood that the built-in light-emitting unit 151, the first camera 152, and the second camera 153 in the head-mounted device 150 are used to perform eye tracking on the user's eyes so that the head-mounted device 150 can interact with the user based on the tracking results.

[0056] The built-in light-emitting unit 151 can be a regular LED light or an infrared (IR) light, etc., and there are no specific restrictions on it here.

[0057] Furthermore, the number of built-in light-emitting units 151 is at least two, and the specific number can be adaptively set according to the eye-tracking accuracy. This application does not impose any specific restrictions on it.

[0058] In other words, at least one built-in light-emitting unit corresponds to the first camera 152, and at least one built-in light-emitting unit corresponds to the second camera 153. This ensures that the first camera 152 and the second camera 153 can capture eye images with light spots, and then perform human-computer interaction operations based on the eye images.

[0059] Considering that the external light-emitting unit 110 is used to assist the electronic device 160 in calibrating the built-in light-emitting unit 151 and camera in the head-mounted device 150, the type of the external light-emitting unit 110 in this application can be determined according to the type of the built-in light-emitting unit 151 on the head-mounted device 150, so that the first camera 152 and the second camera 153 in the head-mounted device 150 can capture images including the external light-emitting unit 110, and thus perform calibration operations based on the images.

[0060] For example, if the built-in light-emitting unit 151 in the head-mounted device 150 is a common LED light, then the external light-emitting unit 110 in this application is a common LED light; if the built-in light-emitting unit 151 in the head-mounted device 150 is an infrared (IR) light, then the external light-emitting unit 110 in this application is an IR light.

[0061] It should be understood that the built-in light-emitting unit 151 and the camera are corresponding. Therefore, when the built-in light-emitting unit 151 is a common LED light, the first camera 152 and the second camera 153 in the head-mounted device 150 can be selected as a common camera or other types of cameras, such as depth cameras. When the built-in light-emitting unit 151 is an IR light, the first camera 152 and the second camera 153 in the head-mounted device 150 can be selected as infrared cameras.

[0062] In this embodiment, the electronic device 160 can be any device with data processing capabilities, such as a computer. The computer can be, but is not limited to, a laptop, desktop computer, or handheld computer.

[0063] In practice, technicians can first fix the external light-emitting unit 110 to the first fixing device 130 and the plane mirror 120 to the second fixing device 140. Then, in the order of the head-mounted device 150, the external light-emitting unit 110, and the plane mirror 120, the external light-emitting unit 110 fixed to the first fixing device 130, the plane mirror 120 fixed to the second fixing device 140, and the head-mounted device 150 are placed on the calibration site or calibration platform, as follows: Figure 2 As shown. When placing the head-mounted device 150, the lens barrel of the head-mounted device 150 is oriented towards the external light-emitting unit 110, and the placement operation is performed based on the ability of the first camera 152 and the second camera 153 to observe the external light-emitting unit 110 and its virtual image in the plane mirror 120. Specifically, the target position is determined where the first camera 152 and the second camera 153 can observe the external light-emitting unit 110 and its virtual image in the plane mirror 120. Then, the head-mounted device 150 is placed at that target position.

[0064] After placing the external light-emitting unit 110, the plane mirror 120, and the head-mounted device 150, the head-mounted device 150 can be activated. Through the established communication connection, the electronic device 160 sends a lighting command to the external light-emitting unit 110 to illuminate it. When the external light-emitting unit 110 is illuminated, the electronic device 160, based on the established communication connection, sends an image acquisition command to the first camera 152 and / or the second camera 153 in the head-mounted device 150. This allows the first camera 152 and / or the second camera 153 to acquire a first image and / or a second image, including the real image of the external light-emitting unit 110 and the virtual image of the external light-emitting unit 110 in the plane mirror 120. Upon receiving the first image sent by the first camera 152 and / or the second image sent by the second camera 153, the electronic device 160 can read the intrinsic and extrinsic parameters of the first camera 152 and the second camera 153 to determine the plane equation of the plane mirror 120 based on the intrinsic and extrinsic parameters of the first camera 152, the intrinsic and extrinsic parameters of the second camera 153, the first image, and / or the second image. In this embodiment, the established communication connection can be achieved through a USB data cable or similar means, and no specific limitations are imposed here.

[0065] It should be noted that the first image and / or the second image can be the first image; or, it can be the second image; or, it can be both the first image and the second image.

[0066] The intrinsic and extrinsic parameters of the first camera 152 and the second camera 153 are configured in the electronic device 160 after the first camera 152 and the second camera 153 have been calibrated. Furthermore, the calibration process for the intrinsic and extrinsic parameters of the first camera 152 and the second camera 153 is a conventional technique in this field and will not be elaborated upon here.

[0067] Furthermore, such as Figure 3As shown, a technician can remove the external light-emitting unit 110, which is fixed to the first fixing device 130, from between the head-mounted device 150 and the plane mirror 120, which is fixed to the second fixing device 140, while keeping the head-mounted device 150 and the plane mirror 120 relatively stationary. Then, an electronic device 160 sends a lighting command to the built-in light-emitting unit 151 in the head-mounted device 150, causing the built-in light-emitting unit 151 to emit light. When the built-in light-emitting unit 151 is lit, the electronic device 160 can send a new image acquisition command to the first camera 152 and the second camera 153, so that the first camera 152 acquires a third image including the virtual image of the built-in light-emitting unit 151 in the plane mirror 120, and / or the second camera 153 acquires a fourth image including the virtual image of the built-in light-emitting unit 151 in the plane mirror 120. When the electronic device 160 receives the third image sent by the first camera 152 and / or the fourth image sent by the second camera 153, it can determine the virtual image coordinates of the built-in light-emitting unit 151 based on the intrinsic and extrinsic parameters of the first camera 152, the intrinsic and extrinsic parameters of the second camera 153, the third image, and / or the fourth image. Then, based on the determined plane equation of the plane mirror 120 and the virtual image coordinates of the built-in light-emitting unit 151, the built-in light-emitting unit 151 is calibrated with the first camera 152 and / or the second camera 153. This achieves the calibration of the built-in light-emitting unit and camera in the head-mounted device 150 based on the external light-emitting unit and the plane mirror, making the entire calibration process simple and quick, thus providing conditions for the batch calibration of head-mounted devices 150 equipped with light-emitting units and cameras.

[0068] The virtual image coordinates refer to the spatial coordinates of the built-in light-emitting unit 151 in the mirror space.

[0069] It is understood that this application calibrates the built-in light-emitting unit 151 and the first camera 152 and / or the second camera 153 based on the determined plane equation of the plane mirror 120 and the virtual image coordinates of the built-in light-emitting unit 151, which may include the following situations:

[0070] Scenario 1

[0071] Based on the plane equation of the plane mirror 120 and the virtual image coordinates of the built-in light-emitting unit 151, the built-in light-emitting unit 151 and the first camera 152, and the built-in light-emitting unit 151 and the second camera 153 are calibrated respectively.

[0072] In other words, this application can perform synchronous calibration operations on the built-in light-emitting unit 151 and the first camera 152, as well as on the built-in light-emitting unit 151 and the second camera 153, based on the determined plane equation and virtual image coordinates.

[0073] Scenario 2

[0074] Based on the plane equation of the plane mirror 120 and the virtual image coordinates of the built-in light-emitting unit 151, the built-in light-emitting unit 151 and the first camera 152 are calibrated.

[0075] Considering that the internal and external parameters and the setting positions of the first camera 151 and the second camera 152 in the head-mounted device 150 are known, this application can calibrate only the built-in light-emitting unit 151 and the first camera 152 based on the plane equation and virtual image coordinates. Then, based on the known parameters, such as the internal and external parameters and setting positions of the cameras, the calibration parameters between the built-in light-emitting unit 151 and the second camera 153 are determined and calibrated based on the calibrated built-in light-emitting unit 151 and the first camera 152. This simplifies the calibration process and further improves calibration speed and efficiency.

[0076] Scenario 3

[0077] Based on the plane equation of the plane mirror 120 and the virtual image coordinates of the built-in light-emitting unit 151, the built-in light-emitting unit 151 and the second camera 153 are calibrated.

[0078] Considering that the internal and external parameters and the setting positions of the first camera 151 and the second camera 152 in the head-mounted device 150 are known, this application can calibrate only the built-in light-emitting unit 151 and the second camera 153 based on the plane equation and virtual image coordinates. Then, based on the known parameters such as the internal and external parameters and setting positions of the cameras, and based on the calibrated built-in light-emitting unit 151 and the second camera 153, the calibration parameters between the built-in light-emitting unit 151 and the first camera 152 are determined and calibrated. This simplifies the calibration process and further improves calibration speed and efficiency.

[0079] The calibration system for the light-emitting unit and camera provided in this application embodiment involves placing an external light-emitting unit fixed to a first fixing device, a plane mirror fixed to a second fixing device, and a head-mounted device in a preset manner. After placement, the electronic device performs the following steps: Based on the intrinsic and extrinsic parameters of the first and second cameras in the head-mounted device, the first image captured by the first camera, and / or the second image captured by the second camera, it determines the plane equation of the plane mirror. Then, based on the intrinsic and extrinsic parameters of the first and second cameras, the third image captured by the first camera, and / or the fourth image captured by the second camera, it determines the virtual image coordinates of the built-in light-emitting unit in the head-mounted device. Finally, based on the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit, it calibrates the built-in light-emitting unit and the first and / or second cameras. This achieves the calibration of the light-emitting unit and camera based on the external light-emitting unit and the plane mirror, making the entire calibration process simple and fast, thus providing conditions for the batch calibration of devices equipped with light-emitting units and cameras.

[0080] After explaining the calibration system for the light-emitting unit and the camera, the calibration method for the light-emitting unit and the camera provided in this application embodiment will be described in detail below based on the above calibration system for the light-emitting unit and the camera.

[0081] It should be noted that the calibration process for the light-emitting unit and the camera in this application is based on the aforementioned calibration system.

[0082] Specifically, such as Figure 4 As shown, the calibration method for the light-emitting unit and the camera provided in this application embodiment may include the following steps:

[0083] S101, acquire a first image captured by the first camera and / or a second image captured by the second camera in the head-mounted device, wherein the first image and / or the second image includes a real image of the external light-emitting unit and a virtual image of the external light-emitting unit in a plane mirror.

[0084] In this embodiment, the type of the external light-emitting unit can be set according to the type of the built-in light-emitting unit in the head-mounted device. For example, if the built-in light-emitting unit in the head-mounted device is a common LED, then the external light-emitting unit in this application is a common LED. As another example, if the built-in light-emitting unit in the head-mounted device is an IR lamp, then the external light-emitting unit in this application is an IR lamp.

[0085] Optionally, after the head-mounted device, the external light-emitting unit, and the plane mirror are placed in the order of head-mounted device, external light-emitting unit, and plane mirror, the technician can activate the head-mounted device and send a lighting command to the external light-emitting unit through the electronic equipment in the calibration system based on the established communication connection, so that the external light-emitting unit lights up. Furthermore, after the external light-emitting unit is normally lit, an image acquisition command can also be sent to the first camera and / or the second camera in the head-mounted device through the electronic equipment in the calibration system based on the established communication connection. This allows the first camera to acquire a first image including the real image of the external light-emitting unit and the virtual image of the external light-emitting unit in the plane mirror, and / or the second camera to acquire a second image including the real image of the external light-emitting unit and the virtual image of the external light-emitting unit in the plane mirror, based on the image acquisition command.

[0086] After the first camera captures the first image and / or the second camera captures the second image, the first camera can send the first image to the electronic device and / or the second camera can send the second image to the electronic device, so as to lay the foundation for the electronic device to determine the plane equation of the plane mirror based on the first image and / or the second image.

[0087] The established communication connection can be achieved through methods such as a USB data cable, and no specific restrictions are imposed here.

[0088] S102, determine the plane equation of the plane mirror based on the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the first image and / or the second image.

[0089] The intrinsic and extrinsic parameters of the first camera and the second camera are calibrated when the first and second cameras are installed in the head-mounted device. Alternatively, the intrinsic and extrinsic parameters of the first and second cameras can be calibrated before the built-in light-emitting unit and camera in the head-mounted device are calibrated. There are no restrictions on this.

[0090] Furthermore, after the first and second cameras are calibrated, the intrinsic and extrinsic parameters of the first and second cameras can be pre-configured in the electronic device to lay the foundation for determining the plane equation of the plane mirror.

[0091] It should be noted that the calibration process of the internal and external parameters of the first and second cameras in this application is a conventional technique in the field, and will not be described in detail here.

[0092] For example, after receiving a first image sent by a first camera and / or a second image sent by a second camera, the electronic device of this application can obtain the intrinsic and extrinsic parameters of the first camera and the intrinsic and extrinsic parameters of the second camera from the configuration information, and then determine the plane equation of the plane mirror based on the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the first image and / or the second image.

[0093] Considering that any plane can be defined using a point on it and its normal vector, this application determines the plane equation of a plane mirror by using the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the first image, and / or the second image to determine the real and virtual image coordinates of the external light-emitting unit. Then, based on the real and virtual image coordinates of the external light-emitting unit, the midpoint coordinates and normal vector used to determine the plane equation of the plane mirror are determined. Finally, using the point-normal form of the plane equation, the plane equation of the plane mirror is determined based on the midpoint coordinates and normal vector.

[0094] Among them, the real image coordinates specifically refer to the spatial coordinates of the external light-emitting unit in the real space; the virtual image coordinates specifically refer to the spatial coordinates of the external light-emitting unit in the mirror space of the plane mirror.

[0095] S103, acquire the third image captured by the first camera and / or the fourth image captured by the second camera, wherein the third image and / or the fourth image includes the virtual image of the built-in light-emitting unit in the head-mounted device in the plane mirror.

[0096] After determining the plane equation of the plane mirror, technicians can remove the external light-emitting unit placed between the head-mounted device and the plane mirror, while keeping the head-mounted device and the plane mirror relatively stationary. Then, through an electronic device based on the established communication connection, a lighting command is sent to the built-in light-emitting unit in the head-mounted device to cause the built-in light-emitting unit to emit light. When it is determined that the built-in light-emitting unit in the head-mounted device is lit, a new image acquisition command can be sent through the electronic device to the first camera and / or the second camera. The first camera, based on the received new image acquisition command, acquires a third image including the virtual image of the built-in light-emitting unit in the plane mirror, and / or the second camera, based on the received new image acquisition command, acquires a fourth image including the virtual image of the built-in light-emitting unit in the plane mirror.

[0097] After the first camera captures the third image and / or the second camera captures the fourth image, the first camera can send the third image to the electronic device and / or the second camera can send the fourth image to the electronic device, so as to lay the foundation for the subsequent electronic device to determine the virtual image coordinates of the built-in light-emitting unit based on the received third and / or fourth images.

[0098] S104, determine the virtual image coordinates of the built-in light-emitting unit based on the internal and external parameters of the first camera, the internal and external parameters of the second camera, the third image and / or the fourth image.

[0099] For example, after receiving the third image sent by the first camera and / or the third image sent by the second camera, this application can determine the virtual image coordinates of the built-in light-emitting unit based on the obtained internal and external parameters of the first camera and the internal and external parameters of the second camera, combined with the third image and / or the fourth image.

[0100] As an optional implementation, when determining the virtual image coordinates of the built-in light-emitting unit, a binocular ranging algorithm can be used to calculate the virtual image coordinates of the built-in light-emitting unit based on the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the third image, and / or the fourth image.

[0101] Determining the virtual image coordinates of the built-in light-emitting unit using a binocular ranging algorithm is a conventional technique in this field, and will not be elaborated on here.

[0102] S105, calibrate the built-in light-emitting unit and the first camera and / or the second camera according to the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit.

[0103] In this embodiment of the application, the calibration of the built-in light-emitting unit with the first camera and / or the second camera specifically includes: calibrating the built-in light-emitting unit with the first camera, and / or calibrating the built-in light-emitting unit with the second camera.

[0104] For example, when calibrating the built-in light-emitting unit with the first camera and / or the second camera, a first calibration parameter between the built-in light-emitting unit and the first camera, and / or a second calibration parameter between the built-in light-emitting unit and the second camera can be determined firstly. Then, the built-in light-emitting unit and the first camera are calibrated according to the first calibration parameter, and / or the built-in light-emitting unit and the second camera are calibrated according to the second calibration parameter.

[0105] Specifically, the first calibration parameter and / or the second calibration parameter refer to the positional relationship between the built-in light-emitting unit and the first camera, and / or the positional relationship between the built-in light-emitting unit and the second camera.

[0106] The calibration method for the light-emitting unit and camera provided in this application determines the plane equation of a plane mirror based on the intrinsic and extrinsic parameters of a first camera and a second camera in a head-mounted device, a first image captured by the first camera, and / or a second image captured by the second camera. Then, based on the intrinsic and extrinsic parameters of the first and second cameras, a third image captured by the first camera, and / or a fourth image captured by the second camera, the virtual image coordinates of the built-in light-emitting unit in the head-mounted device are determined. Finally, the built-in light-emitting unit and the first and / or second cameras are calibrated according to the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit. This achieves the calibration of the light-emitting unit and camera in a head-mounted device based on an external light-emitting unit and a plane mirror, making the entire calibration process simple and quick, thus providing conditions for the batch calibration of head-mounted devices equipped with light-emitting units and cameras.

[0107] As can be seen from the above description, the embodiments of this application calibrate the light-emitting unit and the camera based on the determined plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit.

[0108] Based on the above embodiments, this application further explains how to determine the plane equation of the plane mirror according to the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the first image, and / or the second image, as follows: Figure 5 As shown.

[0109] like Figure 5 As shown, the method may include the following steps:

[0110] S201, acquire a first image captured by a first camera and / or a second image captured by a second camera in the head-mounted device, wherein the first image and / or the second image includes a real image of an external light-emitting unit and a virtual image of the external light-emitting unit in a plane mirror.

[0111] S202, based on the internal and external parameters of the first camera, the internal and external parameters of the second camera, the first image and / or the second image, determine the real image coordinates and virtual image coordinates of the external light-emitting unit.

[0112] S203, determine the plane equation of the plane mirror based on the real image coordinates and virtual image coordinates of the external light-emitting unit.

[0113] In this embodiment, the real image coordinates and virtual image coordinates of the external light-emitting unit are determined by using a binocular ranging algorithm, based on the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the first image, and / or the second image.

[0114] Determining the real and virtual image coordinates of the external light-emitting unit using a binocular ranging algorithm is a conventional technique in this field, and will not be elaborated on here.

[0115] Considering that the midpoint coordinates between the real and virtual image coordinates of the external light-emitting unit lie on the plane of the plane mirror, and that the direction vector determined based on the real and virtual image coordinates of the external light-emitting unit is perpendicular to the normal vector of the plane mirror, this application can determine the midpoint coordinates and the direction vector based on the real and virtual image coordinates of the external light-emitting unit, and then determine the plane equation of the plane mirror based on the determined midpoint coordinates and direction vector.

[0116] For example, the coordinates of the midpoint can be determined using the following formula (1):

[0117]

[0118] Among them, P m P represents the coordinates of the midpoint, P represents the real image coordinates of the external light-emitting unit, and P′ represents the virtual image coordinates of the external light-emitting unit.

[0119] In addition, the direction vector (i.e., the normal vector) can be determined by the following formula (2):

[0120]

[0121] in, Let P represent the normal vector, P' represent the real image coordinates of the external emitting unit, P' represent the virtual image coordinates of the external emitting unit, and ||·|| represent the norm symbol.

[0122] After determining the midpoint coordinates and the normal vector, this application can use the point-normal form of the plane to determine the plane equation of the plane mirror based on the midpoint coordinates and the direction vector.

[0123] S204, acquire a third image captured by the first camera and / or a fourth image captured by the second camera, wherein the third image and / or the fourth image includes a virtual image of the built-in light-emitting unit in the head-mounted device in the plane mirror.

[0124] S205, determine the virtual image coordinates of the built-in light-emitting unit based on the internal and external parameters of the first camera, the internal and external parameters of the second camera, the third image and / or the fourth image.

[0125] S206, calibrate the built-in light-emitting unit and the first camera and / or the second camera according to the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit.

[0126] The calibration method for the light-emitting unit and camera provided in this application determines the plane equation of a plane mirror based on the intrinsic and extrinsic parameters of a first camera and a second camera in a head-mounted device, a first image captured by the first camera, and / or a second image captured by the second camera. Then, based on the intrinsic and extrinsic parameters of the first and second cameras, a third image captured by the first camera, and / or a fourth image captured by the second camera, the virtual image coordinates of the built-in light-emitting unit in the head-mounted device are determined. Finally, the built-in light-emitting unit and the first and / or second cameras are calibrated according to the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit. This achieves the calibration of the light-emitting unit and camera in a head-mounted device based on an external light-emitting unit and a plane mirror, making the entire calibration process simple and quick, thus providing conditions for the batch calibration of head-mounted devices equipped with light-emitting units and cameras.

[0127] Based on the above embodiments, this application further explains how to calibrate the built-in light-emitting unit with the first camera and / or the second camera according to the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit, specifically as follows: Figure 6 As shown.

[0128] like Figure 6 As shown, the method may include the following steps:

[0129] S301, acquire a first image captured by a first camera and / or a second image captured by a second camera in the head-mounted device, wherein the first image and / or the second image includes a real image of an external light-emitting unit and a virtual image of the external light-emitting unit in a plane mirror.

[0130] S302, determine the plane equation of the plane mirror based on the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the first image and / or the second image.

[0131] S303, acquire the third image captured by the first camera and / or the fourth image captured by the second camera, wherein the third image and / or the fourth image includes the virtual image of the built-in light-emitting unit in the head-mounted device in the plane mirror.

[0132] S304, determine the virtual image coordinates of the built-in light-emitting unit based on the internal and external parameters of the first camera, the internal and external parameters of the second camera, the third image and / or the fourth image.

[0133] S305, based on the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit, determine the first calibration parameter between the built-in light-emitting unit and the first camera, and / or the second calibration parameter between the built-in light-emitting unit and the second camera.

[0134] S306, calibrate the built-in light-emitting unit and the first camera according to the first calibration parameters.

[0135] S307, calibrate the built-in light-emitting unit and the second camera according to the second calibration parameters.

[0136] It should be noted that, corresponding to the determination of the first calibration parameter and / or the second calibration parameter in S305, S306 and S307 also have an AND / OR relationship. Specifically, the built-in light-emitting unit and the first camera can be calibrated according to the first calibration parameter; or, the built-in light-emitting unit and the second camera can be calibrated according to the second calibration parameter; or, the built-in light-emitting unit and the first camera can be calibrated according to the first calibration parameter, and simultaneously calibrated according to the second calibration parameter.

[0137] The first calibration parameter refers to the positional relationship between the built-in light-emitting unit and the first camera; the second calibration parameter refers to the positional relationship between the built-in light-emitting unit and the second camera.

[0138] In this embodiment, the positional relationship is specifically an external parameter, and the external parameter represents the real image coordinates of the built-in light-emitting unit.

[0139] In other words, this application determines the physical coordinates of the built-in light-emitting unit based on the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit.

[0140] Considering that the implementation principles and processes for determining the first calibration parameter between the built-in light-emitting unit and the first camera, and / or the second calibration parameter between the built-in light-emitting unit and the second camera are the same or similar, this application will use the determination of the first calibration parameter between the built-in light-emitting unit and the first camera as an example for explanation.

[0141] For example, when determining the first calibration parameter, the virtual image coordinates of the built-in light-emitting unit can be determined, along with the symmetrical point of the plane equation of the plane mirror. Then, the coordinates of this symmetrical point are determined as the first calibration parameter between the built-in light-emitting unit and the first camera. That is, the coordinates of this symmetrical point are determined as the real image coordinates of the built-in light-emitting unit.

[0142] The coordinates of the virtual image of the built-in light-emitting unit, and the point symmetrical about the plane equation of the plane mirror, can be determined in the following way:

[0143] Method 1

[0144] A straight line perpendicular to the plane mirror can be drawn through the virtual image coordinates of the built-in light-emitting unit, and the intersection point of this line and the plane mirror can be determined. Then, the symmetrical point can be determined based on the intersection point.

[0145] The symmetrical point is twice the coordinates of the intersection point.

[0146] Method 2

[0147] Determine the vertical distance between the virtual image coordinates of the built-in light-emitting unit and the plane equation of the plane mirror, and determine the symmetry point based on this vertical distance.

[0148] It should be noted that the above two methods are merely illustrative examples of this application and are not intended to limit this application.

[0149] The calibration method for the light-emitting unit and camera provided in this application determines the plane equation of a plane mirror based on the intrinsic and extrinsic parameters of a first camera and a second camera in a head-mounted device, a first image captured by the first camera, and / or a second image captured by the second camera. Then, based on the intrinsic and extrinsic parameters of the first and second cameras, a third image captured by the first camera, and / or a fourth image captured by the second camera, the virtual image coordinates of the built-in light-emitting unit in the head-mounted device are determined. Finally, the built-in light-emitting unit and the first and / or second cameras are calibrated according to the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit. This achieves the calibration of the light-emitting unit and camera in a head-mounted device based on an external light-emitting unit and a plane mirror, making the entire calibration process simple and quick, thus providing conditions for the batch calibration of head-mounted devices equipped with light-emitting units and cameras.

[0150] The following is a reference to the appendix. Figure 7 This application will now describe a calibration device for a light-emitting unit and a camera, as proposed in an embodiment. Figure 7 This is a schematic block diagram of a calibration device for a light-emitting unit and a camera provided in an embodiment of this application.

[0151] The calibration device 400 for the light-emitting unit and the camera includes: an image acquisition module 410, an equation determination module 420, a coordinate determination module 430, and a calibration module 440.

[0152] The image acquisition module 410 is used to acquire a first image captured by the first camera and / or a second image captured by the second camera in the head-mounted device. The first image and / or the second image include a real image of the external light-emitting unit and a virtual image of the external light-emitting unit in a plane mirror.

[0153] The equation determination module 420 is used to determine the plane equation of the plane mirror based on the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the first image and / or the second image.

[0154] The image acquisition module 410 is further configured to acquire a third image captured by the first camera and / or a fourth image captured by the second camera, wherein the third image and / or the fourth image includes a virtual image of the built-in light-emitting unit in the head-mounted device in the plane mirror;

[0155] The coordinate determination module 430 is used to determine the virtual image coordinates of the built-in light-emitting unit based on the internal and external parameters of the first camera, the internal and external parameters of the second camera, the third image and / or the fourth image.

[0156] The calibration module 440 is used to calibrate the built-in light-emitting unit and the first camera and / or the second camera according to the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit.

[0157] In one optional implementation of this application, the equation determination module 420 includes: a coordinate determination unit and an equation determination unit;

[0158] The coordinate determination unit is used to determine the real image coordinates and virtual image coordinates of the external light-emitting unit based on the internal and external parameters of the first camera, the internal and external parameters of the second camera, the first image and / or the second image.

[0159] The equation determination unit is used to determine the plane equation of the plane mirror based on the real image coordinates and virtual image coordinates of the external light-emitting unit.

[0160] In one optional implementation of this application embodiment, the equation determination unit is specifically used for:

[0161] Determine the midpoint coordinates and direction vector between the real image coordinates and the virtual image coordinates of the external light-emitting unit;

[0162] The plane equation of the plane mirror is determined based on the midpoint coordinates and the direction vector.

[0163] In one optional implementation of this application embodiment, the calibration module 440 includes: a parameter determination unit and a calibration unit;

[0164] The parameter determination unit is used to determine, based on the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit, a first calibration parameter between the built-in light-emitting unit and the first camera, and / or a second calibration parameter between the built-in light-emitting unit and the second camera;

[0165] A calibration unit is configured to calibrate the built-in light-emitting unit and the first camera according to the first calibration parameters; and / or, to calibrate the built-in light-emitting unit and the second camera according to the second calibration parameters.

[0166] In one optional implementation of this application embodiment, the parameter determination unit is specifically used for:

[0167] Determine the virtual image coordinates of the built-in light-emitting unit, and the point symmetrical about the plane equation of the plane mirror;

[0168] The coordinates of the symmetrical point are determined as the calibration parameters between the built-in light-emitting unit and the camera.

[0169] In one optional implementation of this application, the calibration parameter is an external parameter.

[0170] The calibration device for the light-emitting unit and camera provided in this application determines the plane equation of a plane mirror based on the intrinsic and extrinsic parameters of the first and second cameras in the head-mounted device, the first image captured by the first camera, and / or the second image captured by the second camera. It then determines the virtual image coordinates of the built-in light-emitting unit in the head-mounted device based on the intrinsic and extrinsic parameters of the first and second cameras, the third image captured by the first camera, and / or the fourth image captured by the second camera. Finally, it calibrates the built-in light-emitting unit and the first and / or second cameras according to the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit. This achieves the calibration of the light-emitting unit and camera in the head-mounted device based on an external light-emitting unit and a plane mirror, making the entire calibration process simple and quick, thus providing conditions for the batch calibration of head-mounted devices equipped with light-emitting units and cameras.

[0171] It should be understood that the device embodiments and the foregoing method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 7 The device 400 shown can perform Figure 4 The corresponding method embodiments, and the foregoing and other operations and / or functions of each module in device 400 are respectively implemented to achieveFigure 4 For the sake of brevity, the corresponding processes in each method are not described in detail here.

[0172] The apparatus 400 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the first aspect method embodiment in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the first aspect method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the first aspect method embodiment described above.

[0173] Figure 8 This is a schematic block diagram of an electronic device provided in an embodiment of this application.

[0174] like Figure 8 As shown, the electronic device 500 may include:

[0175] The system includes a memory 510 and a processor 520. The memory 510 stores computer programs and transfers the program code to the processor 520. In other words, the processor 520 can retrieve and run the computer program from the memory 510 to implement the calibration method for the light-emitting unit and the camera in this embodiment.

[0176] For example, the processor 520 can be used to execute the above-described video recording method embodiment according to instructions in the computer program.

[0177] In some embodiments of this application, the processor 520 may include, but is not limited to:

[0178] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0179] In some embodiments of this application, the memory 510 includes, but is not limited to:

[0180] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0181] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 510 and executed by the processor 520 to complete the video recording method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device.

[0182] like Figure 8 As shown, the electronic device may also include:

[0183] Transceiver 530, which can be connected to processor 520 or memory 510.

[0184] The processor 520 can control the transceiver 530 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 530 may include a transmitter and a receiver. The transceiver 530 may further include antennas, and the number of antennas may be one or more.

[0185] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0186] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods described in the above method embodiments.

[0187] This application also provides a computer program product containing program instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the above method embodiments.

[0188] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0189] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0191] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0192] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A calibration method for a light-emitting unit and a camera, characterized in that, include: Acquire a first image captured by a first camera and / or a second image captured by a second camera in a head-mounted device, wherein the first image and / or the second image include a real image of an external light-emitting unit and a virtual image of the external light-emitting unit in a plane mirror; The plane equation of the plane mirror is determined based on the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the first image, and / or the second image. After removing the external light-emitting unit, a third image captured by the first camera and / or a fourth image captured by the second camera are obtained, wherein the third image and / or the fourth image includes the virtual image of the built-in light-emitting unit in the head-mounted device in the plane mirror; The virtual image coordinates of the built-in light-emitting unit are determined based on the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the third image, and / or the fourth image. Based on the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit, the built-in light-emitting unit is calibrated with the first camera and / or the second camera.

2. The method according to claim 1, characterized in that, Determining the plane equation of the plane mirror based on the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the first image, and / or the second image includes: Based on the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the first image, and / or the second image, determine the real image coordinates and virtual image coordinates of the external light-emitting unit; The plane equation of the plane mirror is determined based on the real and virtual image coordinates of the external light-emitting unit.

3. The method according to claim 2, characterized in that, The plane equation of the plane mirror is determined based on the real and virtual image coordinates of the external light-emitting unit, including: Determine the midpoint coordinates and direction vector between the real image coordinates and the virtual image coordinates of the external light-emitting unit; The plane equation of the plane mirror is determined based on the midpoint coordinates and the direction vector.

4. The method according to claim 1, characterized in that, The built-in light-emitting unit is calibrated with the first camera and / or the second camera, including: Based on the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit, determine the first calibration parameter between the built-in light-emitting unit and the first camera, and / or the second calibration parameter between the built-in light-emitting unit and the second camera; The built-in light-emitting unit and the first camera are calibrated according to the first calibration parameters; And / or, calibrate the built-in light-emitting unit and the second camera according to the second calibration parameters.

5. The method according to claim 4, characterized in that, Determine the calibration parameters between the built-in light-emitting unit and the camera, including: Determine the virtual image coordinates of the built-in light-emitting unit, and the point symmetrical about the plane equation of the plane mirror; The coordinates of the symmetrical point are determined as the calibration parameters between the built-in light-emitting unit and the camera.

6. The method according to claim 4, characterized in that, The calibration parameters are external parameters.

7. A calibration device for a light-emitting unit and a camera, characterized in that, include: An image acquisition module is used to acquire a first image captured by a first camera and / or a second image captured by a second camera in a head-mounted device. The first image and / or the second image include a real image of an external light-emitting unit and a virtual image of the external light-emitting unit in a plane mirror. The equation determination module is used to determine the plane equation of the plane mirror based on the intrinsic and extrinsic parameters of the first camera, the intrinsic and extrinsic parameters of the second camera, the first image and / or the second image. The image acquisition module is further configured to acquire a third image captured by the first camera and / or a fourth image captured by the second camera after the external light-emitting unit is removed, wherein the third image and / or the fourth image includes the virtual image of the built-in light-emitting unit in the head-mounted device in the plane mirror; The coordinate determination module is used to determine the virtual image coordinates of the built-in light-emitting unit based on the internal and external parameters of the first camera, the internal and external parameters of the second camera, the third image and / or the fourth image; The calibration module is used to calibrate the built-in light-emitting unit and the first camera and / or the second camera based on the plane equation of the plane mirror and the virtual image coordinates of the built-in light-emitting unit.

8. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to perform the calibration method for the light-emitting unit and the camera as described in any one of claims 1 to 6.

9. A calibration system for a light-emitting unit and a camera, characterized in that, include: An external light-emitting unit, a plane mirror, a first fixing device, a second fixing device, a head-mounted device, and the electronic device as described in claim 8; The first fixing device is used to fix the external light-emitting unit; The second fixing device is used to fix the plane mirror; The head-mounted device includes at least a built-in light-emitting unit, a first camera, and a second camera; The electronic device is used to perform the calibration method for the light-emitting unit and the camera as described in any one of 1 to 6 above.

10. A computer-readable storage medium, characterized in that, Used to store computer programs that cause a computer to perform the calibration method for the light-emitting unit and the camera as described in any one of claims 1 to 6.

11. A computer program product containing program instructions, characterized in that, When the program instructions are executed on the electronic device, the electronic device performs the calibration method for the light-emitting unit and the camera as described in any one of claims 1 to 6.

Images