Calibration parameter generation method of augmented reality equipment and electronic equipment

Abstract

The embodiment of the invention relates to the technical field of augmented reality, in particular to a calibration parameter generation method of augmented reality equipment and electronic equipment. The calibration parameter generation method comprises the steps that a first conversion matrix from a real camera coordinate system of a real camera to a virtual camera coordinate system of a virtual camera and a target image acquired by the real camera are acquired, and the target image is acquired by a calibration object placed in an observation environment of augmented reality equipment; a second conversion matrix from the real camera coordinate system to the attitude coordinate system of the augmented reality device is determined, calibration parameters of the augmented reality device are generated according to the first conversion matrix and the second conversion matrix, and the calibration parameters are the conversion matrix from the attitude coordinate system to the virtual camera coordinate system. Therefore, the relative pose between the hardware used for forming the pose coordinate system and the hardware used for projecting the display content to the imaging screen can be calibrated, and it is ensured that the imaging screen normally displays the content.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of augmented reality technology, and in particular to a method for generating calibration parameters of an augmented reality device and an electronic device. Background Art

[0002] AR (Augmented Reality) is a technology that integrates real-world information and virtual-world information for display. Virtual-reality calibration is one of the key technologies for the quality of AR display effects. This technology can more accurately superimpose the content on the screen with objects in the real world. AR devices can display real objects in the real world on the imaging screen of the AR device according to the preset conversion relationship, and display real objects and virtual objects on the imaging screen at the same time, realizing the superimposed display of real objects and virtual objects.

[0003] The relative position and posture between the coordinate system used by the imaging screen of the AR device and a fixed attribute coordinate system on the AR device is an important part of the above-mentioned preset conversion relationship. The accuracy of the relative position and posture directly affects the display effect of the imaging screen. In an AR device, when there is an assembly deviation between the hardware used to project the display content to the imaging screen and the hardware used to form the above-mentioned fixed attribute coordinate system, the relative position and posture between the coordinate systems corresponding to the two hardwares deviates from the design value. If the design value is used as the relative position and posture, it is easy to cause display abnormalities on the imaging screen of the AR device. Summary of the invention

[0004] One purpose of the embodiments of the present application is to provide a calibration parameter generation method, an electronic device and a calibration system for an augmented reality device, so as to solve the technical problem of abnormal imaging screen display caused by assembly deviation between hardware in AR devices in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a method for generating calibration parameters of an augmented reality device, comprising:

[0006] Obtaining a first transformation matrix from a real camera coordinate system of a real camera to a virtual camera coordinate system of a virtual camera and a target image acquired by the real camera, wherein the virtual camera is a camera of the augmented reality device in a virtual space, and the target image is acquired by a calibration object placed in an observation environment of the augmented reality device;

[0007] Determine, according to the target image, a second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device;

[0008] According to the first transformation matrix and the second transformation matrix, calibration parameters of the augmented reality device are generated, and the calibration parameters are the transformation matrix from the posture coordinate system to the virtual camera coordinate system.

[0009] Optionally, determining, according to the target image, a second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device comprises:

[0010] Determine the rotation matrix from the real camera coordinate system to the preset world coordinate system;

[0011] A second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device is determined according to the rotation matrix and the target image.

[0012] Optionally, determining a rotation matrix from a real camera coordinate system to a preset world coordinate system includes:

[0013] Acquire a first relative pose between a calibration object coordinate system of the calibration object and the preset world coordinate system and an intrinsic parameter matrix of the real camera;

[0014] Determining a second relative pose between the calibration object coordinate system and the real camera coordinate system according to the intrinsic parameter matrix of the real camera;

[0015] A rotation matrix from a real camera coordinate system to a preset world coordinate system is determined according to the first relative posture and the second relative posture.

[0016] Optionally, the target image includes multiple groups of sub-image data, and the sub-image data is collected by calibration objects fixedly placed in observation environments at different observation positions of the augmented reality device. The second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device is determined according to the rotation matrix and the target image, including:

[0017] Determine an image segmentation result corresponding to each group of sub-image data;

[0018] A second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device is determined according to the third rotation matrix and the image segmentation result.

[0019] Optionally, the calibration object is placed at a preset angle to the ground, and determining the second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device according to the rotation matrix and the image segmentation result includes:

[0020] According to the preset angle, determining a target constraint matrix;

[0021] A second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device is determined according to the target constraint matrix, the rotation matrix and the image segmentation result.

[0022] Optionally, determining a second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device according to the target constraint matrix, the rotation matrix and the image segmentation result includes:

[0023] Constructing a target equation according to the rotation matrix, the target constraint matrix and the rotation matrix to be solved;

[0024] Solving the rotation matrix to be solved of the target equation according to the image segmentation result to obtain a target rotation matrix;

[0025] A second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device is determined according to the target rotation matrix.

[0026] Optionally, the objective equation is:

[0027] (R world_eye *R glass_eye ·inverse())·inverse()*g=mean_acc i

[0028] Among them, R world_eye is the rotation matrix, R glass_eye is the rotation matrix to be solved, and g is the target constraint matrix;

[0029] Then, solving the rotation matrix to be solved of the target equation according to the image segmentation result to obtain the target rotation matrix includes:

[0030] Substitute the image segmentation result into the mean_acc of the target equation i The rotation matrix to be solved is solved according to a preset optimization algorithm to obtain a target rotation matrix.

[0031] In a second aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is connected to the processor, and the processor is used to execute one or more computer programs stored in the memory, and when the processor executes the one or more computer programs, the electronic device implements the method described above.

[0032] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method as described above.

[0033] In a fourth aspect, an embodiment of the present application provides a calibration system, including:

[0034] Calibration object;

[0035] An augmented reality device, used to observe the calibration object and obtain a target image;

[0036] A real camera, used for capturing the target image;

[0037] The electronic device as described above is communicatively connected with the real camera.

[0038] The embodiments of the present application can achieve the following technical effects: in the calibration parameter generation method of the augmented reality device provided in the embodiments of the present application, it includes: obtaining a first transformation matrix from the real camera coordinate system of the real camera to the virtual camera coordinate system of the virtual camera and a target image captured by the real camera, the virtual camera is the camera of the augmented reality device in the virtual space, the target image is captured by the calibration objects placed in the observation environment of the augmented reality device, according to the target image, determining the second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device, generating the calibration parameters of the augmented reality device according to the first transformation matrix and the second transformation matrix, the calibration parameters are the transformation matrix from the posture coordinate system to the virtual camera coordinate system, therefore, the embodiments of the present application use the real camera as the calibration medium to calibrate the real camera and the augmented reality device respectively. The relative posture between the hardware that forms the posture coordinate system, and the relative posture between the real camera and the hardware in the augmented reality device for projecting display content to the imaging screen, can be calibrated to obtain the relative posture between the hardware in the augmented reality device for forming the posture coordinate system and the hardware in the augmented reality device for projecting display content to the imaging screen. Even if there is an assembly error between the two hardware in the augmented reality device, after calibration by the calibration method provided in the embodiment of the present application, the accurate relative posture between the two hardware can be obtained, thereby avoiding the problem of abnormal display on the imaging screen of the augmented reality device due to the direct use of the relative posture used when the two hardware do not have an assembly error when there is an assembly error between the two hardware, thereby ensuring that the imaging screen can display content normally even when there is an assembly error between the two hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1a and Figure 1b A schematic diagram of the structure of a calibration system provided in an embodiment of the present application;

[0041] Figure 2 for Figure 1a or Figure 1b A structural schematic diagram of an augmented reality device shown;

[0042] Figure 3 A schematic diagram of a flow chart of a method for generating calibration parameters of an augmented reality device provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of the calibration principle of the first conversion matrix provided in an embodiment of the present application;

[0044] Figure 5 It is a schematic diagram of a display screen of an imaging screen provided in an embodiment of the present application;

[0045] Figure 6 yes Figure 3 The schematic diagram of the process of S32 shown;

[0046] Figure 7 is a schematic diagram of an observation position of an augmented reality device provided in an embodiment of the present application;

[0047] Figure 8 It is a structural schematic diagram of a calibration parameter generating device for an augmented reality device provided in an embodiment of the present application;

[0048] Fig. 9 yes Figure 8 The structural diagram of the determination module 82 shown;

[0049] Fig.10 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0051] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional module division is performed in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a sequence different from the module division in the device or the flow chart. Furthermore, the words "first", "second", "third", etc. used in this application do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

[0052] The present application embodiment provides a calibration system. Figure 1a and Figure 1b The calibration system includes an augmented reality device 10, a calibration object 20, a real camera 30 and an electronic device 40.

[0053] The augmented reality device 10 is an electronic device that integrates and displays real-world information and virtual-world information. It can simulate and process physical information that is difficult to experience in the spatial scope of the real world based on computers and other scientific technologies, display virtual information content in the real world for effective application, and superimpose computer-generated virtual models on real scenes to achieve three-dimensional virtual-real fusion display, and interact with users to enhance users' perception of real scenes.

[0054] The augmented reality device 10 can use two-dimensional or three-dimensional objects in the real world as markers to align virtual information with real-world information, that is, the position, size, and movement path of the virtual object perfectly match the real environment, achieving a state where virtual and real coexist. When a user uses the augmented reality device 10 and hopes to display the virtual model in a certain posture more accurately at a certain position in the real scene, a reliable virtual-real calibration method is needed to calibrate the augmented reality device 10, so as to correctly superimpose the virtual information with the objects in the real world, thereby connecting the virtual space with the real scene.

[0055] The augmented reality device 10 may be any type of wearable AR device, such as AR glasses, AR helmets, etc. Among them, AR glasses generally use near-eye display technology, which magnifies the image on the display module through a set of optical systems and projects the image on the retina, so that users can watch the displayed content after wearing it, and users can control the displayed content in the AR glasses by interacting with the AR glasses.

[0056] Since the augmented reality device 10 can be placed anywhere in the real world, it is necessary to select a reference coordinate system in the real world to describe the position of the augmented reality device 10 and use it to describe the position of any object in the real world. This reference coordinate system is referred to as the preset world coordinate system in this article.

[0057] The preset world coordinate system is represented by {world}. In the preset world coordinate system, the position P of any object in the real world w Use coordinates (X w , Y w , Z w ) indicates that the coordinate (X w , Y w , Z w ) is the three-dimensional coordinate of the object in the real world.

[0058] In some embodiments, see Figure 2 The augmented reality device 10 includes an optical component 11, an inertial measurement unit 12 and a controller 13.

[0059] The optical component 11 includes an optical machine and a micro display module. The optical machine is equivalent to a projection device, which can project images onto the micro display module for display. The micro display module can be considered as a translucent imaging screen. On the one hand, the imaging screen is similar to an ordinary eyeglass lens, which can allow the user to see the real world in front of them through external ambient light. On the other hand, the imaging screen can refract or reflect the light emitted by the optical machine to display the virtual image generated by the augmented reality device 10. Therefore, the human eye can see the real world superimposed with the virtual image through the imaging screen.

[0060] The coordinate system used by the imaging screen is the virtual camera coordinate system, where the virtual camera coordinate system is represented by {render}. In addition, a virtual image space coordinate system can be constructed using the imaging screen as a medium. The virtual image space coordinate system is the coordinate system used by the virtual image displayed on the imaging screen, and the virtual image space coordinate system is represented by {virtual}.

[0061] The transformation matrix from the virtual camera coordinate system to the virtual image space coordinate system is denoted by T virtual_render express.

[0062] For a coordinate point P in the virtual camera coordinate system {render} render , coordinate point P render The corresponding coordinate point in the virtual image space coordinate system {virtual} is P virtual , then the coordinate point P virtual It can be expressed as:

[0063] P virtual =T virtual_render *P render

[0064] Among them, T virtual_render It is the transformation matrix from the virtual camera coordinate system {render} to the virtual image space coordinate system {virtual}.

[0065] In some embodiments, the micro display module includes but is not limited to Micro LED or any other type of display module such as MicroOLED, OLED, LCD, etc.

[0066] In some embodiments, the optical component 11 adopts a BirdBath (coaxial air guide) solution, which projects light from the display source to a 45-degree angle beam splitter, allowing the light to be partially reflected and partially transmitted, allowing the user to simultaneously see physical objects in the real world and virtual images generated by the augmented reality device 10.

[0067] The inertial measurement unit 12 is used to sense and track the posture and movement position of the user when using the augmented reality device 10. In this article, the coordinate system used by the inertial measurement unit 12 is the posture coordinate system, which is a coordinate system of fixed attributes of the augmented reality device 10, wherein the posture coordinate system is represented by {glass}.

[0068] For a coordinate point P in the preset world coordinate system {world} w , coordinate point P w The corresponding coordinate point in the posture coordinate system {glass} is P glass , then the coordinate point P glass It can be expressed as:

[0069] P glass =T glass_world *P w

[0070] Among them, T glass_world It is the transformation matrix from the preset world coordinate system {world} to the posture coordinate system {glass}.

[0071] For a coordinate point P in the attitude coordinate system {glass} glass , coordinate point P glass The corresponding coordinate point in the virtual camera coordinate system {render} is P render , then the coordinate point P render It can be expressed as:

[0072] P render =T render_glass *P glass

[0073] Among them, T render_glass It is the transformation matrix from the posture coordinate system {glass} to the virtual camera coordinate system {render}.

[0074] It is understandable that in order to overlay virtual objects with real objects, it is necessary to align the preset world coordinate system {world} with the virtual image space coordinate system {virtual}. There is a fixed conversion relationship T between the preset world coordinate system {world} and the virtual image space coordinate system {virtual}. virtual_world , then T virtual_world It can be expressed as:

[0075] T virtual_world =T virtual_render *T render_glass *T glass_world

[0076] T virtual_render It is a fixed parameter in the augmented reality device 10. The position of the augmented reality device 10 in the preset world coordinate system can be located by the SLAM (Simultaneous Localization and Mapping) algorithm. According to this positioning method, the relative position relationship between the preset world coordinate system and the posture coordinate system of the augmented reality device 10 can be easily calculated, that is, T glass_world , while T render_glass Generally, the design value of the augmented reality device 10 when it leaves the factory can be used. However, when there is a deviation in the assembly between the optical machine and the inertial measurement unit 12, if T render_glass If the design value is still used, the preset world coordinate system {world} and the virtual image space coordinate system {virtual} cannot be aligned, which may easily cause the imaging screen of the augmented reality device 10 to display abnormalities. Therefore, in order to ensure that the preset world coordinate system {world} and the virtual image space coordinate system {virtual} are aligned, an accurate calibration parameter needs to be generated as T render_glass , thereby achieving calibration of the augmented reality device 10.

[0077] The inertial measurement unit 12 includes a gyroscope and an accelerometer, wherein the gyroscope is used to measure the spatial angular velocity of the augmented reality device 10 , and the accelerometer is used to measure the spatial acceleration of the augmented reality device 10 .

[0078] In some embodiments, the gyroscope is a three-axis gyroscope, and the accelerometer is a three-axis accelerometer. The three coordinate axes of the three-axis gyroscope are orthogonal to the three coordinate axes of the three-axis accelerometer, and the coordinate systems of the three-axis gyroscope and the three-axis accelerometer coincide with each other. The three-axis gyroscope can measure the spatial angular velocity around each coordinate axis, and the three-axis accelerometer can measure the spatial acceleration along each coordinate axis.

[0079] The controller 13 is electrically connected to the optical component 11 and the inertial measurement unit 12, respectively, and is used to generate virtual image information and transmit the virtual image information to the optical component 11, so that the micro display module of the optical component 11 displays the virtual image. In addition, the controller 13 calculates the posture angle of the augmented reality device 10 according to the posture and movement position of the user when using the augmented reality device 10, and adjusts the virtual image information according to the posture angle, so that the micro display module of the optical component 11 displays the adjusted virtual image.

[0080] The calibration object 20 is an object placed in the real world to assist in calibrating the augmented reality device 10. The calibration object 20 needs to be placed in front of the augmented reality device 10 to ensure that the augmented reality device 10 can observe the entire calibration object 20 and fully display the target image corresponding to the calibration object 20 on the imaging screen.

[0081] The real camera 30 is a camera placed behind the imaging screen of the augmented reality device 10 to assist in calibrating the augmented reality device 10. Since the augmented reality device 10 usually has two imaging screens, an accurate calibration parameter needs to be generated for each imaging screen. Therefore, two real cameras 30 are needed, and each real camera 30 is used to capture the target image displayed on the corresponding imaging screen.

[0082] Since the method for generating calibration parameters for each imaging screen is similar, the method for generating calibration parameters for the augmented reality device described below is illustrated by taking one of the imaging screens as an example.

[0083] In some embodiments, the real camera 30 is an industrial camera used to replace the human eye, and the optical parameters of the industrial camera are close to those of the human eye, so the industrial camera is called a human eye camera. The human eye camera can well simulate the real effect of the user's glasses viewing the displayed image on the imaging screen.

[0084] The electronic device 40 is communicatively connected with the real camera 30, including but not limited to any type of electronic device such as a mobile phone, a tablet computer, a laptop computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a smart watch, etc. This embodiment does not impose any limitation on the specific type of the electronic device 40.

[0085] The electronic device 40 serves as the algorithm core of the calibration system and is used to execute the calibration parameter generation method of the augmented reality device as described below.

[0086] The present application embodiment provides a method for generating calibration parameters of an augmented reality device. Figure 3 , the calibration parameter generation method of the augmented reality device includes:

[0087] S31, obtaining a first transformation matrix from a real camera coordinate system of a real camera to a virtual camera coordinate system of a virtual camera and a target image acquired by the real camera, wherein the virtual camera is a camera of an augmented reality device in a virtual space, and the target image is acquired by a calibration object placed in an observation environment of the augmented reality device;

[0088] In this step, the virtual camera is a camera that does not exist in the augmented reality device. As mentioned earlier, the light emitted by the optical machine will be reflected and displayed on the imaging screen. Based on similar imaging principles, this process can be simulated into the imaging process of a camera, and the imaging process can be considered to be completed by the virtual camera.

[0089] The real camera coordinate system of the real camera is represented by {eye}, and the first transformation matrix from the real camera coordinate system {eye} to the virtual camera coordinate system {render} is represented by T render_eye express.

[0090] The target image is acquired by collecting calibration objects placed in the augmented reality device observation environment, and then the target image is acquired by photographing the imaging screen of the augmented reality device with a real camera. The target image acquired by the real camera is then processed by an electronic device to calibrate and obtain a first transformation matrix.

[0091] When calibrating the first transformation matrix, refer to Figure 4 The electronic device can construct a calibration method through the plane {eye_screen} of the target image acquired by the real camera and the imaging screen {render_screen}, and based on the target image, generate a first transformation matrix from the real camera coordinate system {eye} to the virtual camera coordinate system {render}.

[0092] S32. Determine a second transformation matrix from a real camera coordinate system to a posture coordinate system of an augmented reality device according to the target image;

[0093] In this step, the second transformation matrix from the real camera coordinate system {eye} to the posture coordinate system {glass} of the augmented reality device is expressed as T glass_eye .

[0094] See also Figure 5 , Figure 5 T on the left render_glass When the user turns his head left and right, he sees a virtual image on the imaging screen. Figure 5 T on the right render_glass In the case of inaccuracy, the user sees a virtual image on the imaging screen when turning his head left and right. Since the assembly error between the inertial measurement unit and the optical machine is usually at the millimeter level, such as Figure 5 As shown in the figure, the assembly error can be regarded as a ray. No matter how much the translation is, the error caused by the ray is consistent. Therefore, when there is an assembly error between the inertial measurement unit and the optical machine, the translation vector t from the real camera coordinate system {eye} to the attitude coordinate system {glass} is glass_eye The effect on the display on the imaging screen is minimal, however, even a small rotation angle may have a significant effect on the display on the imaging screen.

[0095] Since the translation vector t from the real camera coordinate system {eye} to the attitude coordinate system {glass} glass_eye The display of the imaging screen has little effect, so the translation vector t from the attitude coordinate system {glass} to the real camera coordinate system {eye} glass_eye The factory design value of the augmented reality device can be used directly, and the second conversion matrix T glass_eye The rotation matrix R from the real camera coordinate system {eye} to the attitude coordinate system {glass} glass_eye And the translation vector t from the real camera coordinate system {eye} to the attitude coordinate system {glass} glass_eye Therefore, in the calibration of the second transformation matrix T glass_eye When , we only need to calibrate the rotation matrix R from the real camera coordinate system {eye} to the attitude coordinate system {glass} glass_eye That's it.

[0096] S33, according to the first conversion matrix T render_eye And the second transformation matrix T glass_eye , generate the calibration parameters of the augmented reality device. The calibration parameters are the transformation matrix from the posture coordinate system {glass} to the virtual camera coordinate system {render}. The transformation matrix from the posture coordinate system {glass} to the virtual camera coordinate system {render} is expressed by the following formula:

[0097] T render_glass =T render_eye *T eye_glass

[0098] Among them, T render_glass is the transformation matrix from the pose coordinate system {glass} to the virtual camera coordinate system {render}, T render_eye is the first transformation matrix, T eye_glass This is the result of taking the inverse of the second transformation matrix.

[0099] Therefore, the embodiment of the present application uses a real camera as a calibration medium to calibrate the relative posture between the real camera and the inertial measurement unit, and the relative posture between the real camera and the optical machine, respectively, so as to calibrate the relative posture between the inertial measurement unit and the optical machine. Even if there is an assembly error between the inertial measurement unit and the optical machine, after calibration by the calibration method provided in the embodiment of the present application, the accurate relative posture between the inertial measurement unit and the optical machine can be obtained, thereby avoiding the problem of abnormal display on the imaging screen of the augmented reality device due to the direct use of the relative posture used when there is no assembly error between the inertial measurement unit and the optical machine when there is an assembly error between the inertial measurement unit and the optical machine, thereby ensuring that the imaging screen can display content normally even when there is an assembly error between the inertial measurement unit and the optical machine.

[0100] In some embodiments, see Figure 6 , S22 includes:

[0101] S321, determining the rotation matrix from the real camera coordinate system {eye} to the preset world coordinate system {world};

[0102] In this step, the rotation matrix from the real camera coordinate system {eye} to the preset world coordinate system {world} is R world_eye express.

[0103] In some embodiments, first, the electronic device obtains a first relative pose between a calibration object coordinate system and a preset world coordinate system {world} and an intrinsic parameter matrix of a real camera.

[0104] The calibration object coordinate system is represented by {apriltag}, and the first relative position between the calibration object coordinate system {apriltag} and the preset world coordinate system {world} is represented by T world_apriltag Indicates that the first relative posture T world_apriltag It can be determined by the assembly characteristics of the calibration object. For example, if the calibration object is a plate-shaped calibration object, when the plate-shaped calibration object is perpendicular to the ground, the first relative position T world_apriltag It can be determined as:

[0105]

[0106] Among them, R world_apriltag It is the rotation matrix from the calibration object coordinate system {apriltag} to the preset world coordinate system {world}.

[0107] The intrinsic parameter matrix of the real camera is K eye express.

[0108] Next, the electronic device determines a second relative posture between the calibration object coordinate system {apriltag} and the real camera coordinate system {eye} according to the intrinsic parameter matrix of the real camera.

[0109] The second relative pose of the calibration object coordinate system {apriltag} and the real camera coordinate system {eye} is expressed by T apriltag_eye The electronic device can use any appropriate apriltag recognition algorithm, and use K according to the intrinsic parameter matrix of the real camera eye Calculate the second relative pose T between the calibration object coordinate system {apriltag} and the real camera coordinate system {eye} apriltag_eye , T apriltag_eye =[R apriltag_eye |t apriltag_eye ], where R apriltag_eyeis the rotation matrix from the real camera coordinate system {eye} to the calibration object coordinate system {apriltag}, t apriltag_eye is the translation vector from the real camera coordinate system {eye} to the calibration object coordinate system {apriltag}.

[0110] Finally, the electronic device determines the rotation matrix of the real camera coordinate system {eye} and the preset world coordinate system {world} according to the first relative posture and the second relative posture, wherein the rotation matrix of the real camera coordinate system {eye} and the preset world coordinate system {world} is expressed by the following formula:

[0111] R world_eye =R world_apriltag *R apriltag_eye

[0112] Among them, R world_eye is the rotation matrix between the real camera coordinate system {eye} and the preset world coordinate system {world}, R world_apriltag is the rotation matrix between the calibration object coordinate system {apriltag} and the preset world coordinate system {world}, R apriltag_eye It is the rotation matrix from the real camera coordinate system {eye} to the calibration object coordinate system {apriltag}.

[0113] S322: Determine the second transformation matrix T from the real camera coordinate system {eye} to the posture coordinate system {glass} of the augmented reality device according to the rotation matrix and the target image. glass_eye .

[0114] In some embodiments, the target image includes multiple groups of sub-image data, and the sub-image data are collected by the augmented reality device using calibration objects that are fixedly placed in the observation environment at different observation positions.

[0115] For example, see Figure 7 , the calibration object is fixed on the ground, and the augmented reality device is Figure 6 The positions 1, 2, 3, 4 and 5 shown in the figure change the observation positions in turn. When the augmented reality device is placed at position 1 to observe the calibration object for a preset time, the augmented reality device collects the target image of the calibration object for a preset time, and the real camera collects a group of sub-image data for a preset time. Then, when the augmented reality device is placed at position 2 to observe the calibration object for a preset time, the augmented reality device collects the target image of the calibration object for a preset time, and the real camera collects another group of sub-image data for a preset time... Thus, the augmented reality device is operated according to Figure 7 After the positions 1, 2, 3, 4 and 5 shown in the figure change the observation positions in sequence, the real camera collects 5 groups of sub-image data with a preset duration.

[0116] The preset time length can be set according to actual needs. For example, the preset time length is 1 second.

[0117] In some embodiments, the electronic device determines an image segmentation result corresponding to each group of sub-image data, and determines a second transformation matrix from the virtual camera coordinate system to the posture coordinate system of the augmented reality device based on the rotation matrix and the image segmentation result.

[0118] For example, the image segmentation result is the acc mean of the sub-image data, where acc refers to the ratio of the number of correctly classified pixels in the segmentation result to the total number of pixels when the image is segmented. The electronic device can compare the predicted label of each pixel with the true label and calculate the accuracy between them. Specifically, the number of pixels in the segmentation result that are the same as the true result can be counted, and then divided by the total number of pixels to obtain the accuracy of the segmentation. As mentioned above, the electronic device can use any appropriate mean algorithm to calculate the acc mean_acc of the 5 groups of sub-image data respectively. i (i∈[0,4]).

[0119] In some embodiments, the calibration object is placed at a preset angle to the ground, and the electronic device can determine the target constraint matrix according to the preset angle, and determine the second transformation matrix T from the virtual camera coordinate system {render} to the posture coordinate system {glass} according to the target constraint matrix, the rotation matrix and the image segmentation result. glass_eye .

[0120] This embodiment introduces the target constraint matrix to improve the electronic device's determination of the second transformation matrix T from the virtual camera coordinate system {render} to the posture coordinate system {glass}. glass_eye efficiency and accuracy.

[0121] In some embodiments, the preset angle is 90 degrees, that is, the calibration object is placed perpendicular to the ground.

[0122] When the calibration object is placed perpendicular to the ground, the target constraint matrix is ​​[0 0 9.8] T , where [0 0 9.8] T Also the value of gravity.

[0123] In some embodiments, the electronic device constructs a target equation according to the rotation matrix, the target constraint matrix and the rotation matrix to be solved, solves the rotation matrix to be solved of the target equation according to the image segmentation result, obtains the target rotation matrix, and determines the second transformation matrix T from the real camera coordinate system {eye} to the posture coordinate system {glass} according to the target rotation matrix. glass_eye .

[0124] In this embodiment, the rotation matrix to be solved is the unknown matrix of the target equation. After solving the unknown matrix, the matrix obtained is the rotation matrix R from the real camera coordinate system {eye} to the attitude coordinate system {glass} glass_eye , which is the target rotation matrix.

[0125] As mentioned before, the translation vector t from the pose coordinate system {glass} to the real camera coordinate system {eye} eye_glass The design value of the augmented reality device when it leaves the factory can be used directly. Therefore, the electronic device rotates according to the target rotation matrix R glass_eye and the known translation vector t eye_glass , the second transformation matrix T can be determined glass_eye .

[0126] In some embodiments, the objective equation is expressed as follows:

[0127] (R world_eye *R glass_eye ·inverse())·inverse()*g=mean_acc i

[0128] Among them, R world_eye is the rotation matrix from the real camera coordinate system {eye} to the preset world coordinate system {world}, R glass_eye is the rotation matrix from the real camera coordinate system {eye} to the posture coordinate system {glass}, g is the target constraint matrix, and inverse() represents the inverse operation.

[0129] In some embodiments, the electronic device may substitute the image segmentation result corresponding to each group of sub-image data into the mean_acc i And solve the rotation matrix of the target equation according to the preset optimization algorithm to obtain the target rotation matrix R glass_eye .

[0130] For example, as mentioned above, the electronic device can substitute the image segmentation results corresponding to the five groups of sub-image data into the mean_acc of the target equation respectively. i , and use optimization algorithms such as LM (Levenberg-Marquardt) algorithm as the preset optimization algorithm to iteratively optimize the target equation. The rotation matrix to be solved after the iterative optimization is finally calculated as the target rotation matrix R glass_eye .

[0131] It should be noted that, in each of the above-mentioned embodiments, there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present application, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0132] As another aspect of the embodiment of the present application, the embodiment of the present application provides a calibration parameter generation device for an augmented reality device. The calibration parameter generation device for an augmented reality device can be a software module, and the software module includes a number of instructions, which are stored in a memory, and the processor can access the memory and call the instructions for execution to complete the calibration parameter generation method for the augmented reality device described in the above-mentioned various embodiments.

[0133] In some embodiments, the calibration parameter generation device of the augmented reality device can also be constructed by hardware devices. For example, the calibration parameter generation device of the augmented reality device can be constructed by one or more chips, and each chip can work in coordination with each other to complete the calibration parameter generation method of the augmented reality device described in the above embodiments. For another example, the calibration parameter generation device of the augmented reality device can also be constructed by various logic devices, such as a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0134] In some embodiments, see Figure 8 The calibration parameter generation device 800 of the augmented reality device includes an acquisition module 81, a determination module 82 and a generation module 83.

[0135] The acquisition module 81 is used to obtain the first transformation matrix from the real camera coordinate system of the real camera to the virtual camera coordinate system of the virtual camera and the target image captured by the real camera. The virtual camera is the camera of the augmented reality device in the virtual space, and the target image is captured by the calibration objects placed in the observation environment of the augmented reality device. The determination module 82 is used to determine the second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device according to the target image. The generation module 83 is used to generate the calibration parameters of the augmented reality device according to the first transformation matrix and the second transformation matrix. The calibration parameters are the transformation matrix from the posture coordinate system to the virtual camera coordinate system.

[0136] Therefore, the embodiment of the present application uses a real camera as a calibration medium to calibrate the relative posture between the real camera and the inertial measurement unit, and the relative posture between the real camera and the optical machine, respectively, so as to calibrate the relative posture between the inertial measurement unit and the optical machine. Even if there is an assembly error between the inertial measurement unit and the optical machine, after calibration by the calibration method provided in the embodiment of the present application, the accurate relative posture between the inertial measurement unit and the optical machine can be obtained, thereby avoiding the problem of abnormal display on the imaging screen of the augmented reality device due to the direct use of the relative posture used when there is no assembly error between the inertial measurement unit and the optical machine when there is an assembly error between the inertial measurement unit and the optical machine, thereby ensuring that the imaging screen can display content normally even when there is an assembly error between the inertial measurement unit and the optical machine.

[0137] In some embodiments, see Fig. 9 The determination module 82 includes a first determination unit 821 and a second determination unit 822 .

[0138] The first determination unit 821 is used to determine the rotation matrix of the real camera coordinate system and the preset world coordinate system, and the second determination unit 822 is used to determine the second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device according to the rotation matrix and the target image.

[0139] In some embodiments, the first determination unit 821 is specifically used to: obtain a first relative pose between the calibration object coordinate system and the preset world coordinate system and the intrinsic parameter matrix of the real camera, determine a second relative pose between the calibration object coordinate system and the real camera coordinate system according to the intrinsic parameter matrix of the real camera, and determine a rotation matrix from the real camera coordinate system to the preset world coordinate system according to the first relative pose and the second relative pose.

[0140] In some embodiments, the target image includes multiple groups of sub-image data, which are collected by calibration objects fixedly placed in the observation environment at different observation positions of the augmented reality device. The second determination unit 822 is specifically used to: determine the image segmentation result corresponding to each group of sub-image data, and determine the second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device based on the rotation matrix and the image segmentation result.

[0141] It should be noted that the calibration parameter generation device of the above-mentioned augmented reality device can execute the calibration parameter generation method of the augmented reality device provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in the embodiment of the calibration parameter generation device of the augmented reality device, please refer to the calibration parameter generation method of the augmented reality device provided in the embodiment of the present application.

[0142] See also Fig.10 , Fig.104 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 40 includes one or more processors 41 and a memory 42. The memory 42 is connected to the one or more processors, for example, connected to the processors via a bus.

[0143] The processor 41 is configured to support the electronic device 40 to perform the corresponding functions in the method in the above method embodiment. The processor can be a central processing unit (CPU), a network processor (NP), a hardware chip or any combination thereof. The above hardware chip can be an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0144] The memory 42 is used to store program codes, etc. The memory 42 may include a volatile memory (VM), such as a random access memory (RAM); the memory may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 42 may also include a combination of the above-mentioned types of memory.

[0145] The memory 42 may be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the calibration parameter generation method of the augmented reality device in the embodiment of the present application. The processor 41 executes various functional applications and data processing of the calibration parameter generation method of the augmented reality device and the calibration parameter generation device of the augmented reality device by running the non-volatile software programs, instructions and modules stored in the memory, that is, to realize the functions of each module or unit of the calibration parameter generation method of the augmented reality device and the calibration parameter generation device of the augmented reality device provided in the above method embodiment.

[0146] The memory 42 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function. The data storage area may store data created according to the use of the calibration parameter generation device of the augmented reality device, etc. In some embodiments, the memory 42 may optionally include a memory remotely arranged relative to the processor 41, and these remote memories may be connected to the calibration parameter generation device of the augmented reality device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] The one or more modules are stored in the memory 42, and when executed by the one or more processors 41, the calibration parameter generation method of the augmented reality device in any of the above-mentioned method embodiments is executed, for example, the method steps described in the above-mentioned method embodiments are executed to realize the functions of the modules described in the above-mentioned device embodiments.

[0148] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method described in the above embodiment.

[0149] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0150] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for generating calibration parameters of an augmented reality device, characterized in that: include: Obtaining a first transformation matrix from a real camera coordinate system of a real camera to a virtual camera coordinate system of a virtual camera and a target image acquired by the real camera, wherein the virtual camera is a camera of the augmented reality device in a virtual space, and the target image is acquired by a calibration object placed in an observation environment of the augmented reality device; Determine, according to the target image, a second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device; According to the first transformation matrix and the second transformation matrix, calibration parameters of the augmented reality device are generated, and the calibration parameters are the transformation matrix from the posture coordinate system to the virtual camera coordinate system.

2. The method according to claim 1, characterized in that: Determining, according to the target image, a second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device comprises: Determine the rotation matrix from the real camera coordinate system to the preset world coordinate system; A second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device is determined according to the rotation matrix and the target image.

3. The method according to claim 2, characterized in that Determining the rotation matrix from the real camera coordinate system to the preset world coordinate system includes: Acquire a first relative pose between a calibration object coordinate system of the calibration object and the preset world coordinate system and an intrinsic parameter matrix of the real camera; Determining a second relative pose between the calibration object coordinate system and the real camera coordinate system according to the intrinsic parameter matrix of the real camera; A rotation matrix from a real camera coordinate system to a preset world coordinate system is determined according to the first relative posture and the second relative posture.

4. The method according to claim 2, characterized in that: The target image includes multiple groups of sub-image data, and the sub-image data is collected by the augmented reality device through calibration objects fixedly placed in the observation environment at different observation positions. The second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device is determined according to the rotation matrix and the target image, including: Determine an image segmentation result corresponding to each group of sub-image data; A second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device is determined according to the rotation matrix and the image segmentation result.

5. The method according to claim 4, characterized in that The calibration object is placed at a preset angle to the ground, and the second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device is determined according to the rotation matrix and the image segmentation result, including: According to the preset angle, determining a target constraint matrix; A second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device is determined according to the target constraint matrix, the rotation matrix and the image segmentation result.

6. The method according to claim 5, characterized in that Determining the second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device according to the target constraint matrix, the rotation matrix and the image segmentation result includes: Constructing a target equation according to the rotation matrix, the target constraint matrix and the rotation matrix to be solved; Solving the rotation matrix to be solved of the target equation according to the image segmentation result to obtain a target rotation matrix; A second transformation matrix from the real camera coordinate system to the posture coordinate system of the augmented reality device is determined according to the target rotation matrix.

7. The method according to claim 6, characterized in that The objective equation is: (R world_eye *R glass_eye ·inverse())·inverse()*g=mean_acc i Among them, R world_eye is the rotation matrix, R glass_eye is the rotation matrix to be solved, and g is the target constraint matrix; Then, solving the rotation matrix to be solved of the target equation according to the image segmentation result to obtain the target rotation matrix includes: Substitute the image segmentation result into the mean_acc of the target equation i The rotation matrix to be solved is solved according to a preset optimization algorithm to obtain a target rotation matrix.

8. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is connected to the processor, and the processor is used to execute one or more computer programs stored in the memory, and when the processor executes the one or more computer programs, the electronic device implements the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

10. A calibration system, characterized in that: include: Calibration object; An augmented reality device, used to observe the calibration object and obtain a target image; A real camera, used for capturing the target image; The electronic device as claimed in claim 8, wherein the electronic device is communicatively connected to the real camera.

Images