Device initialization method, device, electronic device, and computer-readable storage medium

Abstract

The embodiments of the present application disclose a device initialization method, apparatus, electronic device, and computer-readable storage medium. The method includes: in response to a startup operation for initializing inertial vision parameters, obtaining inertial information of an extended reality device and collecting scene images through the extended reality device; identifying feature point information and scene plane information of the scene image; determining the inertial vision parameters and target plane coefficients of the extended reality device based on the plane coefficients to be solved, inertial information, and feature point information of the scene plane information, wherein the target plane coefficients are the solutions corresponding to the plane coefficients to be solved. Inertial vision solution constraints based on the plane coefficients to be solved are implemented to improve the accuracy of the solved inertial vision parameters. After the inertial vision parameter initialization is completed, the solution of the plane coefficients to be solved is simultaneously realized, thereby improving the efficiency of environmental understanding and the efficiency of interaction.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of extended reality technology, and specifically to a device initialization method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] Extended Reality (XR) is a comprehensive term encompassing virtual reality (VR), augmented reality (AR), and mixed reality (MR). XR technology expands human perception and interaction, creating virtual, augmented, and mixed reality experiences through digital technology interacting with the real world. XR devices are widely used in scenarios such as navigation, information display, and augmented reality experiences.

[0003] Visual-Inertial Odometry (VIO) is a key positioning and tracking algorithm in the field of extended reality. It achieves high-precision positioning of devices in three-dimensional space by fusing data from visual and inertial sensors.

[0004] However, to improve the accuracy of inertial visual odometry, it needs to be initialized. However, the current initialization process lacks reference to environmental information, and the accuracy of the initialization results needs to be improved. In addition, environmental understanding is usually only possible after initialization, which affects interaction efficiency. Summary of the Invention

[0005] The embodiments of the present application provide a device initialization method, apparatus, electronic device, and computer-readable storage medium, which can improve the accuracy of inertial vision initialization results and improve interaction efficiency.

[0006] In a first aspect, an embodiment of the present application provides a device initialization method, applied to an extended reality device, the method comprising:

[0007] In response to a start-up operation of initializing inertial vision parameters, acquiring inertial information of the extended reality device and capturing a scene image through the extended reality device;

[0008] Identifying feature point information and scene plane information of the scene image;

[0009] The inertial visual parameters and the target plane coefficient of the extended reality device are determined according to the plane coefficient to be solved of the scene plane information, the inertial information and the feature point information, wherein the target plane coefficient is the solution corresponding to the plane coefficient to be solved.

[0010] In a second aspect, an embodiment of the present application further provides a device initialization apparatus, which is applied to an extended reality device, and the apparatus includes:

[0011] a response module, configured to obtain inertial information of the extended reality device and capture a scene image through the extended reality device in response to a startup operation of initializing inertial vision parameters;

[0012] A recognition module, configured to recognize feature point information and scene plane information of the scene image;

[0013] A determination module is used to determine the inertial visual parameters and the target plane coefficient of the extended reality device based on the plane coefficient to be solved of the scene plane information, the inertial information and the feature point information, wherein the target plane coefficient is the solution corresponding to the plane coefficient to be solved.

[0014] Optionally, in some embodiments of the present application, determining the inertial visual parameters and the target plane coefficient of the extended reality device based on the to-be-solved plane coefficient of the scene plane information, the inertial information, and the feature point information includes:

[0015] Determine the three-dimensional coordinate representation corresponding to the feature point information according to the plane coefficient to be solved of the scene plane information;

[0016] Inertial visual parameters and a target plane coefficient of the extended reality device are determined according to the three-dimensional coordinate representation, the inertial information, and the feature point information.

[0017] Optionally, in some embodiments of the present application, determining the three-dimensional coordinate representation corresponding to the feature point information according to the plane coefficient to be solved of the scene plane information includes:

[0018] Determining a spatial plane expression model corresponding to the scene plane information, where the spatial plane expression model is constructed based on the plane coefficients to be solved for the scene plane information;

[0019] A three-dimensional coordinate representation is determined based on the image coordinate representation of the feature point information corresponding to the scene plane information, the camera internal parameters corresponding to the extended reality device, and the spatial plane expression model.

[0020] Optionally, in some embodiments of the present application, determining the three-dimensional coordinate representation based on the image coordinate representation of the feature point information corresponding to the scene plane information, the camera intrinsic parameters corresponding to the extended reality device, and the spatial plane expression model includes:

[0021] Converting the image coordinate representation of the feature point information corresponding to the scene plane information into a normalized coordinate representation according to the intrinsic camera parameters corresponding to the extended reality device;

[0022] The three-dimensional coordinate representation is determined according to the conversion relationship between the normalized coordinate representation and the three-dimensional coordinate representation and the spatial plane expression model.

[0023] Optionally, in some embodiments of the present application, the inertial information is acquired based on an inertial measurement unit in the extended reality device, the scene image is acquired by an image acquisition module in the extended reality device, and the three-dimensional coordinate representation includes a three-dimensional representation of the feature point information in the coordinate system of the image acquisition module;

[0024] The determining of the inertial vision parameters and the target plane coefficient of the extended reality device according to the three-dimensional coordinate representation, the inertial information, and the feature point information includes:

[0025] Determining a three-dimensional coordinate conversion relationship and a three-dimensional coordinate projection relationship, wherein the three-dimensional coordinate conversion relationship includes a conversion relationship between a first coordinate system and a second coordinate system at the same time, and the three-dimensional coordinate projection relationship includes a conversion relationship between the first coordinate system and the second coordinate system at different times, wherein the first coordinate system corresponds to the inertial measurement unit, and the second coordinate system corresponds to the image acquisition module;

[0026] Constructing a target model to be solved according to the three-dimensional coordinate representation, the three-dimensional coordinate transformation relationship and the three-dimensional coordinate projection relationship;

[0027] The target model to be solved is solved according to the inertial information and the feature point information, and the inertial visual parameters and the target plane coefficient of the extended reality device are determined.

[0028] Optionally, in some embodiments of the present application, constructing the target model to be solved according to the three-dimensional coordinate representation, the three-dimensional coordinate transformation relationship, and the three-dimensional coordinate projection relationship includes:

[0029] Determining a first three-dimensional representation from the three-dimensional coordinate representation, where the first three-dimensional representation is a three-dimensional representation of the feature point information at the initial moment of the start-up operation in a second coordinate system at the initial moment of the image acquisition module;

[0030] determining a second three-dimensional representation based on the first three-dimensional representation and the three-dimensional coordinate conversion relationship, wherein the second three-dimensional representation represents a three-dimensional representation of a feature point at an initial moment of the startup operation in the first coordinate system of the inertial measurement unit;

[0031] A target model to be solved is constructed according to the second three-dimensional representation and the three-dimensional coordinate projection relationship.

[0032] Optionally, in some embodiments of the present application, determining the three-dimensional coordinate projection relationship includes:

[0033] Determining an inertial vision observation model corresponding to the inertial measurement unit;

[0034] Performing pre-integration processing on the inertial visual observation model to obtain a pre-integration representation of inertial parameters;

[0035] Determining a three-dimensional coordinate transformation relationship for the inertial measurement unit according to the inertial parameter pre-integration representation, the three-dimensional coordinate transformation relationship including a transformation relationship of the coordinate system of the feature point information at different times;

[0036] A three-dimensional coordinate projection relationship is constructed according to the three-dimensional coordinate transformation relationship and the external parameters between the inertial measurement unit and the image acquisition module.

[0037] Optionally, in some embodiments of the present application, solving the target model to be solved based on the inertial information and the feature point information to determine the inertial visual parameters and target plane coefficients of the extended reality device includes:

[0038] Transforming the target model to be solved according to the coupling relationship to obtain a simplified model, wherein the coupling relationship includes a coupling relationship between the plane coefficient to be solved and the inertial information to be solved;

[0039] The simplified model is solved according to the inertial information and the feature point information to obtain the inertial vision parameters of the inertial measurement unit at the initial moment of the startup operation and the target plane coefficient of the scene image at the initial moment.

[0040] Optionally, in some embodiments of the present application, the inertial vision parameters include initial inertial vision parameters at the initial moment of the startup operation, and the target plane coefficients include initial plane coefficients of scene plane information at the initial moment;

[0041] After determining the inertial visual parameters and the target plane coefficient of the extended reality device according to the to-be-solved plane coefficient of the scene plane information, the inertial information, and the feature point information, the method further includes:

[0042] Determine the three-dimensional coordinates of the feature point information at the initial moment according to the initial plane coefficients;

[0043] Solving a nonlinear optimization model according to the three-dimensional coordinates, the initial inertial vision parameters, the inertial information, and the feature point information to obtain optimized plane coefficients corresponding to the scene plane information at the initial moment and the optimized inertial vision parameters at the initial moment;

[0044] The nonlinear optimization model is constructed based on a spatial plane expression model corresponding to a three-dimensional coordinate projection relationship, a pre-integrated representation of inertia parameters, and scene plane information at an initial moment.

[0045] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps in the above-mentioned device initialization method are implemented.

[0046] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned device initialization method are implemented.

[0047] In a fifth aspect, embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in the embodiments of the present application.

[0048] The extended reality device of an embodiment of the present application responds to a startup operation of initializing inertial visual parameters, obtains inertial information of the extended reality device, and captures a scene image through the extended reality device, identifies feature point information and scene plane information of the scene image, and determines the inertial visual parameters and target plane coefficients of the extended reality device based on the plane coefficients to be solved, inertial information, and feature point information of the scene plane information, wherein the target plane coefficients are the solution corresponding to the plane coefficients to be solved.

[0049] Among them, the embodiment of the present application initializes the inertial vision parameters by combining the plane coefficients to be solved with the scene plane information, realizes the inertial vision solution constraints based on the plane coefficients to be solved, and improves the accuracy of the solved inertial vision parameters.

[0050] Among them, by initializing the inertial vision parameters in combination with the plane coefficients to be solved, the plane coefficients to be solved are solved after the inertial vision parameter initialization is completed, and the plane detection is completed, so that the plane detection no longer depends on the completion of the initialization of the inertial vision parameters, thereby improving the efficiency of environmental understanding and the interaction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 1 is a schematic diagram of a scenario in which an extended reality device according to an embodiment of the present application executes the device initialization method;

[0053] Figure 2 This is a flow chart of a device initialization method provided in an embodiment of the present application;

[0054] Figure 3 This is a schematic diagram of the structure of the device initialization apparatus provided in an embodiment of the present application;

[0055] Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the present application.

[0056] Description of Figure Numbers:

[0057] 10-extended reality device; 201-response module; 202-identification module; 203-determination module; 301-processor; 302-memory; 303-power supply; 304-input unit. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0059] The embodiments of the present application provide a device initialization method, apparatus, electronic device, and computer-readable storage medium. Specifically, the embodiments of the present application provide a device initialization apparatus applicable to electronic devices, which is used to improve the accuracy of inertial vision initialization results and improve interaction efficiency. Specifically, the electronic device includes a terminal device with an inertial measurement unit and an image capture module, the terminal device including but not limited to an extended reality device, the extended reality device including but not limited to a head-mounted display, wearable glasses, and the terminal device may also be an integrated terminal device with a built-in computing and processing unit, or a split terminal device with an external computing and processing unit. The terminal device includes but is not limited to an airborne optical display system, i.e., a head-up display system, used on vehicles such as aircraft, automobiles, and ships, for example, an AR-HUD (Augmented Reality HUD-up Display) installed on a smart connected car, extended reality applications (such as extended reality games, virtual tourism, telemedicine, or virtual experiments) used on handheld mobile devices such as mobile phones, laptops, and tablets, and near-eye display systems used on wearable devices such as head-mounted displays and smart glasses.

[0060] See also Figure 1 , Figure 1 : is a schematic diagram of a scenario in which an extended reality device according to an embodiment of the present application executes the device initialization method. The specific execution process of the device initialization method executed by the extended reality device is as follows:

[0061] In response to navigation and positioning tasks, the extended reality device 10 initiates an operation to initialize inertial vision parameters. In response to the operation, the extended reality device 10 obtains inertial information in a short period of time and collects a scene image. Subsequently, the extended reality device identifies the scene image to obtain feature point information and scene plane information. Based on the plane coefficients to be solved of the scene plane information, the inertial information and the feature point information, the inertial vision parameters of the extended reality device and the target plane coefficients for the plane coefficients to be solved are determined, thereby initializing the inertial vision parameters of the extended reality device.

[0062] Subsequently, the extended reality device can use the initialized inertial vision parameters to perform navigation and positioning tasks, and at the same time, can also use the target plane coefficient to perform extended reality interaction processing.

[0063] In summary, the embodiments of the present application initialize the inertial vision parameters by combining the plane coefficients to be solved with the scene plane information, thereby realizing the inertial vision solution constraints based on the plane coefficients to be solved and improving the accuracy of the solved inertial vision parameters.

[0064] Among them, by initializing the inertial vision parameters in combination with the plane coefficients to be solved, the plane coefficients to be solved are solved after the inertial vision parameter initialization is completed, and the plane detection is completed, so that the plane detection no longer depends on the completion of the initialization of the inertial vision parameters, thereby improving the efficiency of environmental understanding and the interaction efficiency.

[0065] It should be noted that the order of description of the following embodiments does not limit the priority order of the embodiments.

[0066] See also Figure 2 , Figure 2 A flowchart of a device initialization method provided in an embodiment of the present application. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the flowchart. Specifically, the execution subject of the device initialization method is an extended reality device. Specifically, the process of the extended reality device executing the device initialization method specifically includes:

[0067] 101. In response to a startup operation of initializing inertial vision parameters, obtain inertial information of the extended reality device and capture a scene image through the extended reality device.

[0068] It is understandable that the startup operation is the first startup operation when applying the inertial vision parameters or the inertial vision system, and the purpose is to complete the initialization of the inertial vision parameters at the first time so as to perform high-precision positioning based on the initialized inertial vision parameters.

[0069] Inertial information is data collected through visual and inertial sensors, such as an inertial measurement unit (IMU). This inertial information includes acceleration information, angular velocity information, etc. It is understood that to improve the accuracy of inertial vision parameter initialization, this inertial information or inertial data is collected shortly after the initialization operation is initiated.

[0070] Among them, the scene image is an image that matches the inertial information and is collected within the aforementioned short period of time. For example, the scene image is collected by an image acquisition module (such as a camera) on the extended reality device while collecting inertial information based on the inertial measurement unit.

[0071] 102. Identify feature point information and scene plane information of the scene image.

[0072] Feature point information is information that is characteristic or key in a scene image, also known as key point information. Feature point information includes the location of the feature point, usually expressed as coordinates.

[0073] The scene plane information refers to the planes contained in the scene image and information related to the planes, such as the number of planes, the size of the planes, and the positions of the planes.

[0074] It is understandable that the feature point information can be extracted from the scene image based on a deep learning model or a machine learning model; the scene plane information can be obtained by performing plane segmentation on the scene image through a lightweight plane semantic segmentation network. It is understandable that each scene image may include multiple planes.

[0075] 103. Determine the inertial visual parameters and target plane coefficients of the extended reality device based on the plane coefficients to be solved of the scene plane information, the inertial information, and the feature point information, wherein the target plane coefficients are solutions corresponding to the plane coefficients to be solved.

[0076] It should be noted that the plane coefficient is a coefficient that reflects the characteristics of the plane, for example, the plane coefficient includes the plane normal vector and the plane distance. The plane coefficient to be solved is the plane coefficient to be solved corresponding to the scene plane information in the scene image.

[0077] In summary, the embodiments of the present application initialize the inertial vision parameters by combining the plane coefficients to be solved with the scene plane information, thereby realizing the inertial vision solution constraints based on the plane coefficients to be solved and improving the accuracy of the solved inertial vision parameters.

[0078] Among them, by initializing the inertial vision parameters in combination with the plane coefficients to be solved, the plane coefficients to be solved are solved after the inertial vision parameter initialization is completed, and the plane detection is completed, so that the plane detection no longer depends on the completion of the initialization of the inertial vision parameters, thereby improving the efficiency of environmental understanding and the interaction efficiency.

[0079] The accuracy of inertial vision parameter initialization can be improved based on the three-dimensional coordinates corresponding to the feature point information, wherein the three-dimensional coordinates of the feature point information can be represented based on the plane coefficients of the scene plane information where the feature point information is located. That is, optionally, in some embodiments of the present application, the step of "determining the inertial vision parameters and the target plane coefficients of the extended reality device based on the plane coefficients to be solved of the scene plane information, the inertial information, and the feature point information" includes:

[0080] Determine the three-dimensional coordinate representation corresponding to the feature point information according to the plane coefficient to be solved of the scene plane information;

[0081] Inertial visual parameters and a target plane coefficient of the extended reality device are determined according to the three-dimensional coordinate representation, the inertial information, and the feature point information.

[0082] Among them, the inertial vision parameters are initialized by using a three-dimensional coordinate representation based on the plane coefficient to be solved, so that when the initialization is completed, the target plane coefficient corresponding to the plane coefficient to be solved is directly solved, thereby realizing the synchronous solution of the plane coefficient when the inertial vision parameters are initialized.

[0083] Among them, the three-dimensional coordinate representation of the feature point information can be determined based on the spatial plane expression model. That is, optionally, in some embodiments of the present application, the step of "determining the three-dimensional coordinate representation corresponding to the feature point information based on the plane coefficient to be solved of the scene plane information" includes:

[0084] Determining a spatial plane expression model corresponding to the scene plane information, where the spatial plane expression model is constructed based on the plane coefficients to be solved for the scene plane information;

[0085] A three-dimensional coordinate representation is determined based on the image coordinate representation of the feature point information corresponding to the scene plane information, the camera internal parameters corresponding to the extended reality device, and the spatial plane expression model.

[0086] Furthermore, the normalized coordinates corresponding to the feature point information may be determined based on the camera internal parameters of the augmented reality device, and the three-dimensional coordinate representation may be determined based on the normalized coordinates.

[0087] For example, use Represents each frame of scene image and the coordinate system of each frame of scene image, where, Represents the first frame scene image (also represents the coordinate system of the first frame scene image), that is, the scene image collected at the initial moment of the startup operation, and the feature point information (i.e., feature point) on the scene image is used express, Representing scene images The kth feature point information or feature point extracted from The three-dimensional coordinates in the coordinate system are expressed as , and the spatial plane expression model corresponding to the plane in space is: , where the normal vector of the plane is expressed as , D represents the plane distance. Correspondingly, for the above feature point information, the spatial plane expression model is satisfied, that is, the following expression is satisfied:

[0088] .

[0089] Among them, the camera intrinsic parameters of the extended reality device are expressed as , feature point information The corresponding normalized coordinates are , then there is always:

[0090] , .

[0091] Based on the above and , We can get:

[0092]

[0093] in, , , .

[0094] Then, the three-dimensional coordinate representation of the feature point information based on the plane coefficient to be solved is obtained, that is:

[0095]

[0096] That is, through the above processing, the representation of the three-dimensional coordinates by the plane coefficients to be solved based on the scene plane information is achieved.

[0097] In an embodiment of the present application, a target model to be solved based on inertial visual parameters and plane coefficients to be solved can be established, and the inertial visual parameters and target plane coefficients can be determined by solving the target model to be solved. In this embodiment, since the inertial information is acquired based on the inertial measurement unit, and the scene image is acquired based on the image acquisition module, the coordinate systems of the inertial information and the scene image are different. Therefore, it is necessary to combine the coordinate transformation relationship to construct the target model to be solved, that is, optionally, in some embodiments of the present application, the inertial information is acquired based on the inertial measurement unit in the extended reality device, and the scene image is acquired by the image acquisition module in the extended reality device. The three-dimensional coordinate representation includes the three-dimensional representation of the feature point information in the coordinate system of the image acquisition module. The step of "determining the inertial visual parameters and target plane coefficients of the extended reality device based on the three-dimensional coordinate representation, the inertial information and the feature point information" includes:

[0098] Determining a three-dimensional coordinate conversion relationship and a three-dimensional coordinate projection relationship, wherein the three-dimensional coordinate conversion relationship includes a conversion relationship between a first coordinate system and a second coordinate system at the same time, and the three-dimensional coordinate projection relationship includes a conversion relationship between the first coordinate system and the second coordinate system at different times, wherein the first coordinate system corresponds to the inertial measurement unit, and the second coordinate system corresponds to the image acquisition module;

[0099] Constructing a target model to be solved according to the three-dimensional coordinate representation, the three-dimensional coordinate transformation relationship and the three-dimensional coordinate projection relationship;

[0100] The target model to be solved is solved according to the inertial information and the feature point information, and the inertial visual parameters and the target plane coefficient of the extended reality device are determined.

[0101] Among them, since the purpose of initializing the inertial measurement parameters is to obtain the inertial measurement parameters at the initial moment of the startup operation, the three-dimensional coordinate transformation relationship and the three-dimensional coordinate projection relationship can be established based on the three-dimensional coordinate representation at the initial moment. That is, optionally, in some embodiments of the present application, the step of "constructing the target model to be solved according to the three-dimensional coordinate representation, the three-dimensional coordinate transformation relationship and the three-dimensional coordinate projection relationship" includes:

[0102] Determining a first three-dimensional representation from the three-dimensional coordinate representation, where the first three-dimensional representation is a three-dimensional representation of the feature point information at the initial moment of the start-up operation in a second coordinate system at the initial moment of the image acquisition module;

[0103] determining a second three-dimensional representation based on the first three-dimensional representation and the three-dimensional coordinate conversion relationship, wherein the second three-dimensional representation represents a three-dimensional representation of a feature point at an initial moment of the startup operation in the first coordinate system of the inertial measurement unit;

[0104] A target model to be solved is constructed according to the second three-dimensional representation and the three-dimensional coordinate projection relationship.

[0105] Among them, the three-dimensional coordinate projection relationship can be obtained by pre-integration processing based on the inertial visual observation model. That is, optionally, in some embodiments of the present application, the step of "determining the three-dimensional coordinate projection relationship" includes:

[0106] Determining an inertial vision observation model corresponding to the inertial measurement unit;

[0107] Performing pre-integration processing on the inertial visual observation model to obtain a pre-integration representation of inertial parameters;

[0108] Determining a three-dimensional coordinate transformation relationship for the inertial measurement unit according to the inertial parameter pre-integration representation, the three-dimensional coordinate transformation relationship including a transformation relationship of the coordinate system of the feature point information at different times;

[0109] A three-dimensional coordinate projection relationship is constructed according to the three-dimensional coordinate transformation relationship and the external parameters between the inertial measurement unit and the image acquisition module.

[0110] For example, the first three-dimensional representation is recorded as: , the second three-dimensional representation is recorded as: , a second three-dimensional representation based on the first three-dimensional representation and the three-dimensional coordinate transformation relationship Then record it as:

[0111]

[0112] Among them, the above The formula of is recorded as the first formula, and Represents the external parameters between the inertial measurement unit and the image acquisition module. The coordinate system of the inertial measurement unit is recorded as , the three-dimensional coordinate projection relationship is to transform the coordinate system of the inertial measurement unit To the coordinate system of the image acquisition module Therefore, the transformation based on the three-dimensional coordinate projection relationship is Towards The conversion is expressed as:

[0113]

[0114] Among them, the above The formula of is recorded as the second formula, and Represents the external parameters between the image acquisition module and the inertial measurement unit, Represents the coordinate system and The external parameters between Indicates that The feature point information in the coordinate system is converted into The feature point information in the coordinate system is The feature point information projection in the coordinate system is Feature point information in the coordinate system.

[0115] Among them, for The feature point information in the coordinate system is converted into The feature point information in the coordinate system can be obtained based on the pre-integration method. For example, for the inertial measurement unit, the inertial information collected includes accelerometer data and gyroscope data , accordingly, the inertial visual inertial model corresponding to the inertial measurement unit includes:

[0116]

[0117] in, , is a random deviation, is the expression of gravity acceleration in the inertial world coordinate system. At the same time, the accelerometer and gyroscope measurements are also affected by white noise, which are Through Euler median integral, the position and velocity of two adjacent moments can be connected by the following formula:

[0118] in, Represents the attitude, corresponding to the rotation matrix from the world coordinate system G to the inertial measurement unit coordinate system B, Represents the velocity in the world coordinate system G, Indicates the position in the world coordinate system G.

[0119] When the state of the inertial vision measurement system changes, the observation data of the inertial measurement unit will be recalculated based on the above relationship. In order to avoid repeated calculations, the above relationship is sorted out using the pre-integration method, which gives:

[0120]

[0121] in, is a pre-integrated term that is only affected by the random bias.

[0122] Similarly, based on the above pre-integration design, we can get the i-th frame, that is, arrive The pre-integration between , that is, including: Similarly, we can also get the representation between the first frame and the i-th frame, that is, arrive The pre-integration between , that is, including: .

[0123] Furthermore, since only the three-dimensional coordinate representation of the first frame is required, based on the above pre-integration representation, we can obtain:

[0124]

[0125] Furthermore, assume that:

[0126]

[0127] Based on The pre-integral representation and the above assumptions can be sorted out to obtain the third formula:

[0128]

[0129] Then, the first and third formulas are substituted into the second formula to obtain the target model to be solved, which is expressed as:

[0130]

[0131] From the above formula, we can see that the only unknown variable is the plane coefficient. And the speed and gravity in the coordinate system The expression below , .

[0132] Further assume that:

[0133]

[0134] Then, based on , It can be seen that Therefore, multiply both the left and right sides of the target model to be solved by , in order to eliminate the above-mentioned target model to be solved , and the transformed target model is obtained, which is expressed as:

[0135]

[0136] Further assume that:

[0137]

[0138] Based on the above assumptions, we can get:

[0139]

[0140] For the above formula, multiply both sides of the equation by , then put the known quantity on the right side of the equation and the part with the unknown quantity on the left side of the equation, and continue to simplify to get the fourth formula, then we have:

[0141]

[0142] Among them, it can be seen from the above formula that the plane coefficient and inertia information or inertia coefficient , will be coupled with each other, so we can solve based on the coupling relationship, that is, optionally, in some embodiments of the present application, the step of "solving the target model to be solved according to the inertial information and the feature point information, and determining the inertial visual parameters and target plane coefficients of the extended reality device" includes:

[0143] Transforming the target model to be solved according to the coupling relationship to obtain a simplified model, wherein the coupling relationship includes a coupling relationship between the plane coefficient to be solved and the inertial information to be solved;

[0144] The simplified model is solved according to the inertial information and the feature point information to obtain the inertial vision parameters of the inertial measurement unit at the initial moment of the startup operation and the target plane coefficient of the scene image at the initial moment.

[0145] For example, the mutually coupled parameters are treated as unknown quantities again, so the assumption (referred to as coupling assumption) is:

[0146]

[0147] Based on the above assumptions, the fourth formula is sorted out to obtain:

[0148]

[0149] Arranging the above formula into a linear matrix expression, we have: .in,

[0150]

[0151]

[0152]

[0153] Furthermore, from the linear matrix expression It can be seen that a feature point information in the scene image A 2*1 linear constraint will be generated. Assuming that there are K feature point information on a plane and a total of L scene images, a total of 2*K*L equality constraints will be generated. As long as 2KL≥21, the above linear matrix expression We can get the solution, that is, we can get the above If we get The initial value of the solution can be indirectly solved by the above coupling assumptions. and , and based on , , The relationship between the target plane coefficients corresponding to the plane coefficients to be solved is obtained. , and .

[0154] After solving the inertial visual parameters, the gravitational acceleration can be obtained. The value below , assuming that the magnitude of gravity in the inertial world coordinate system is , through Schmidt orthogonal decomposition, we can obtain , express The rotation in the inertial world coordinate system is simultaneously get The velocity in the inertial world coordinate system, as for the position Can be directly set to zero vector. Based on the obtained coordinate system Rotation in world coordinates ,Location ,speed , we can recover each one according to the above two adjacent moments’ posture and velocity formulas The pose and velocity are expressed as: , and .

[0155] Furthermore, after solving the inertial visual parameters and target plane coefficients at the initial moment, the inertial visual parameters, target plane coefficients, and A nonlinear optimization model is established based on the posture and speed of the augmented reality device, and then the inertial vision parameters and the target plane parameters at the initial moment are optimized. That is, optionally, in some embodiments of the present application, the inertial vision parameters include the initial inertial vision parameters at the initial moment of the startup operation, and the target plane coefficients include the initial plane coefficients of the scene plane information at the initial moment. The step of "determining the inertial vision parameters and the target plane coefficients of the augmented reality device according to the plane coefficients to be solved of the scene plane information, the inertial information, and the feature point information" includes:

[0156] Determine the three-dimensional coordinates of the feature point information at the initial moment according to the initial plane coefficients;

[0157] Solving a nonlinear optimization model according to the three-dimensional coordinates, the initial inertial vision parameters, the inertial information, and the feature point information to obtain optimized plane coefficients corresponding to the scene plane information at the initial moment and the optimized inertial vision parameters at the initial moment;

[0158] The nonlinear optimization model is constructed based on a spatial plane expression model corresponding to a three-dimensional coordinate projection relationship, a pre-integrated representation of inertia parameters, and scene plane information at an initial moment.

[0159] For example, the variables to be optimized are:

[0160]

[0161] in, .

[0162] A nonlinear optimization model is established, which is expressed as:

[0163]

[0164] in, Represents the reprojection error constraint equation, corresponding to the formula:

[0165]

[0166] and

[0167]

[0168] in, Represents the inertial measurement unit (IMU) pre-integration constraint equation, corresponding to the formula:

[0169]

[0170] in, Represents the plane constraint equation, corresponding formula:

[0171] .

[0172] Among them, the nonlinear optimization library (ceres) is used to Solve and obtain more accurate optimized plane coefficients and optimized inertial vision parameters, for example, obtain the posture and speed at each moment, and then obtain the trajectory, and also obtain the three-dimensional coordinates of each feature point information.

[0173] In summary, the embodiments of the present application initialize the inertial vision parameters by combining the plane coefficients to be solved with the scene plane information, thereby realizing the inertial vision solution constraints based on the plane coefficients to be solved and improving the accuracy of the solved inertial vision parameters.

[0174] Among them, by initializing the inertial vision parameters in combination with the plane coefficients to be solved, the plane coefficients to be solved are solved after the inertial vision parameter initialization is completed, and the plane detection is completed, so that the plane detection no longer depends on the completion of the initialization of the inertial vision parameters, thereby improving the efficiency of environmental understanding and the interaction efficiency.

[0175] To facilitate better implementation of the device initialization method of the present application, the present application also provides a device initialization apparatus based on the above device initialization method. The meanings of the terms are the same as those in the above device initialization method, and the specific implementation details can be referred to the description in the method embodiment.

[0176] See also Figure 3 , Figure 3 : is a schematic diagram of the structure of a device initialization device provided in an embodiment of the present application. The device initialization device is applied to an extended reality device. The device initialization device can be specifically as follows:

[0177] A response module 201 is configured to obtain inertial information of the extended reality device and capture a scene image through the extended reality device in response to a startup operation of initializing inertial vision parameters;

[0178] Identification module 202, used to identify feature point information and scene plane information of the scene image;

[0179] The determination module 203 is used to determine the inertial visual parameters and the target plane coefficient of the extended reality device based on the plane coefficient to be solved of the scene plane information, the inertial information and the feature point information, wherein the target plane coefficient is the solution corresponding to the plane coefficient to be solved.

[0180] Optionally, in some embodiments of the present application, determining the inertial visual parameters and the target plane coefficient of the extended reality device based on the to-be-solved plane coefficient of the scene plane information, the inertial information, and the feature point information includes:

[0181] Determine the three-dimensional coordinate representation corresponding to the feature point information according to the plane coefficient to be solved of the scene plane information;

[0182] Inertial visual parameters and a target plane coefficient of the extended reality device are determined according to the three-dimensional coordinate representation, the inertial information, and the feature point information.

[0183] Optionally, in some embodiments of the present application, determining the three-dimensional coordinate representation corresponding to the feature point information according to the plane coefficient to be solved of the scene plane information includes:

[0184] Determining a spatial plane expression model corresponding to the scene plane information, where the spatial plane expression model is constructed based on the plane coefficients to be solved for the scene plane information;

[0185] A three-dimensional coordinate representation is determined based on the image coordinate representation of the feature point information corresponding to the scene plane information, the camera internal parameters corresponding to the extended reality device, and the spatial plane expression model.

[0186] Optionally, in some embodiments of the present application, determining the three-dimensional coordinate representation based on the image coordinate representation of the feature point information corresponding to the scene plane information, the camera intrinsic parameters corresponding to the extended reality device, and the spatial plane expression model includes:

[0187] Converting the image coordinate representation of the feature point information corresponding to the scene plane information into a normalized coordinate representation according to the intrinsic camera parameters corresponding to the extended reality device;

[0188] The three-dimensional coordinate representation is determined according to the conversion relationship between the normalized coordinate representation and the three-dimensional coordinate representation and the spatial plane expression model.

[0189] Optionally, in some embodiments of the present application, the inertial information is acquired based on an inertial measurement unit in the extended reality device, the scene image is acquired by an image acquisition module in the extended reality device, and the three-dimensional coordinate representation includes a three-dimensional representation of the feature point information in the coordinate system of the image acquisition module;

[0190] The determining of the inertial vision parameters and the target plane coefficient of the extended reality device according to the three-dimensional coordinate representation, the inertial information, and the feature point information includes:

[0191] Determining a three-dimensional coordinate conversion relationship and a three-dimensional coordinate projection relationship, wherein the three-dimensional coordinate conversion relationship includes a conversion relationship between a first coordinate system and a second coordinate system at the same time, and the three-dimensional coordinate projection relationship includes a conversion relationship between the first coordinate system and the second coordinate system at different times, wherein the first coordinate system corresponds to the inertial measurement unit, and the second coordinate system corresponds to the image acquisition module;

[0192] Constructing a target model to be solved according to the three-dimensional coordinate representation, the three-dimensional coordinate transformation relationship and the three-dimensional coordinate projection relationship;

[0193] The target model to be solved is solved according to the inertial information and the feature point information, and the inertial visual parameters and the target plane coefficient of the extended reality device are determined.

[0194] Optionally, in some embodiments of the present application, constructing the target model to be solved according to the three-dimensional coordinate representation, the three-dimensional coordinate transformation relationship, and the three-dimensional coordinate projection relationship includes:

[0195] Determining a first three-dimensional representation from the three-dimensional coordinate representation, where the first three-dimensional representation is a three-dimensional representation of the feature point information at the initial moment of the start-up operation in a second coordinate system at the initial moment of the image acquisition module;

[0196] determining a second three-dimensional representation based on the first three-dimensional representation and the three-dimensional coordinate conversion relationship, wherein the second three-dimensional representation represents a three-dimensional representation of a feature point at an initial moment of the startup operation in the first coordinate system of the inertial measurement unit;

[0197] A target model to be solved is constructed according to the second three-dimensional representation and the three-dimensional coordinate projection relationship.

[0198] Optionally, in some embodiments of the present application, determining the three-dimensional coordinate projection relationship includes:

[0199] Determining an inertial vision observation model corresponding to the inertial measurement unit;

[0200] Performing pre-integration processing on the inertial visual observation model to obtain a pre-integration representation of inertial parameters;

[0201] Determining a three-dimensional coordinate transformation relationship for the inertial measurement unit according to the inertial parameter pre-integration representation, the three-dimensional coordinate transformation relationship including a transformation relationship of the coordinate system of the feature point information at different times;

[0202] A three-dimensional coordinate projection relationship is constructed according to the three-dimensional coordinate transformation relationship and the external parameters between the inertial measurement unit and the image acquisition module.

[0203] Optionally, in some embodiments of the present application, solving the target model to be solved based on the inertial information and the feature point information to determine the inertial visual parameters and target plane coefficients of the extended reality device includes:

[0204] Transforming the target model to be solved according to the coupling relationship to obtain a simplified model, wherein the coupling relationship includes a coupling relationship between the plane coefficient to be solved and the inertial information to be solved;

[0205] The simplified model is solved according to the inertial information and the feature point information to obtain the inertial vision parameters of the inertial measurement unit at the initial moment of the startup operation and the target plane coefficient of the scene image at the initial moment.

[0206] Optionally, in some embodiments of the present application, the inertial vision parameters include initial inertial vision parameters at the initial moment of the startup operation, and the target plane coefficients include initial plane coefficients of scene plane information at the initial moment;

[0207] After determining the inertial visual parameters and the target plane coefficient of the extended reality device according to the to-be-solved plane coefficient of the scene plane information, the inertial information, and the feature point information, the method further includes:

[0208] Determine the three-dimensional coordinates of the feature point information at the initial moment according to the initial plane coefficients;

[0209] Solving a nonlinear optimization model according to the three-dimensional coordinates, the initial inertial vision parameters, the inertial information, and the feature point information to obtain optimized plane coefficients corresponding to the scene plane information at the initial moment and the optimized inertial vision parameters at the initial moment;

[0210] The nonlinear optimization model is constructed based on a spatial plane expression model corresponding to a three-dimensional coordinate projection relationship, a pre-integrated representation of inertia parameters, and scene plane information at an initial moment.

[0211] In an embodiment of the present application, the response module 201 obtains the inertial information of the extended reality device and captures a scene image through the extended reality device in response to a startup operation of initializing the inertial vision parameters. The recognition module 202 identifies the feature point information and scene plane information of the scene image. The determination module 203 determines the inertial vision parameters and the target plane coefficient of the extended reality device based on the plane coefficient to be solved of the scene plane information, the inertial information, and the feature point information, wherein the target plane coefficient is the solution corresponding to the plane coefficient to be solved.

[0212] In summary, the embodiments of the present application initialize the inertial vision parameters by combining the plane coefficients to be solved with the scene plane information, thereby realizing the inertial vision solution constraints based on the plane coefficients to be solved and improving the accuracy of the solved inertial vision parameters.

[0213] Among them, by initializing the inertial vision parameters in combination with the plane coefficients to be solved, the plane coefficients to be solved are solved after the inertial vision parameter initialization is completed, and the plane detection is completed, so that the plane detection no longer depends on the completion of the initialization of the inertial vision parameters, thereby improving the efficiency of environmental understanding and the interaction efficiency.

[0214] In addition, the present application also provides an electronic device, such as Figure 4 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0215] The electronic device may include one or more processors 301 of processing cores, one or more computer-readable storage media memories 302, a power supply 303, an input unit 304 and other components. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0216] The processor 301 is the control center of the electronic device. It connects all parts of the electronic device using various interfaces and lines. By running or executing software programs and / or modules stored in the memory 302 and accessing data stored in the memory 302, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 301.

[0217] Memory 302 can be used to store software programs and modules. Processor 301 executes various functional applications and device initialization by running the software programs and modules stored in memory 302. Memory 302 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as sound playback or image playback); the data storage area may store data generated based on the use of the electronic device. Memory 302 may also include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory 302 may also include a memory controller to provide processor 301 with access to memory 302.

[0218] The electronic device also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 303 can also include one or more DC or AC power supplies, a recharging system, a power supply device debugging circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0219] The electronic device may further include an input unit 304, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0220] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 301 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, thereby implementing the steps of any device initialization method provided in the embodiments of the present application.

[0221] In an embodiment of the present application, the electronic device (terminal device) may also be an extended reality device, which measures the distance of the object to be measured using the inertial measurement module and camera module on the extended reality device. The extended reality device includes smart glasses in the form of natural glasses, which have at least the following functions: extended reality display, wearing status detection, biometric recognition, human-computer interaction, and data processing. The smart glasses include a frame, temples, a processor, sensors, an optical display assembly, a microphone, a speaker, etc. The temples and frame contain cavities within which circuits and electronic components are placed. Sensors include cameras, eye trackers, iris meters, IMUs, gyroscopes, etc. The optical display assembly includes a micro-projection engine and an optical coupler. The micro-projection engine can be based on Micro-OLEDs, Micro-LEDs, LCOS, or LBS. The optical coupler can be an optical lens or an optical waveguide. The processor can be a specialized XR processor or a general-purpose processor. The frame and temples are the supporting structure of the entire glasses. The temples have a certain degree of elasticity, and the length and clamping force can be adjusted to suit users with different head shapes; the camera can capture the user's hands, face, eyes, etc., the microphone can listen to the user's voice, and the processor can calculate and process various data.

[0222] The extended reality device of an embodiment of the present application responds to a startup operation of initializing inertial visual parameters, obtains inertial information of the extended reality device, and captures a scene image through the extended reality device, identifies feature point information and scene plane information of the scene image, and determines the inertial visual parameters and target plane coefficients of the extended reality device based on the plane coefficients to be solved, inertial information, and feature point information of the scene plane information, wherein the target plane coefficients are the solution corresponding to the plane coefficients to be solved.

[0223] Among them, the embodiment of the present application initializes the inertial vision parameters by combining the plane coefficients to be solved with the scene plane information, realizes the inertial vision solution constraints based on the plane coefficients to be solved, and improves the accuracy of the solved inertial vision parameters.

[0224] Among them, by initializing the inertial vision parameters in combination with the plane coefficients to be solved, the plane coefficients to be solved are solved after the inertial vision parameter initialization is completed, and the plane detection is completed, so that the plane detection no longer depends on the completion of the initialization of the inertial vision parameters, thereby improving the efficiency of environmental understanding and the interaction efficiency.

[0225] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0226] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0227] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any device initialization method provided in the present application.

[0228] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0229] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0230] Since the instructions stored in the computer-readable storage medium can execute the steps in any device initialization method provided in the present application, the beneficial effects that can be achieved by any device initialization method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0231] The above is a detailed introduction to a device initialization method, apparatus, electronic device, and computer-readable storage medium provided in this application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A device initialization method, characterized in that: Applied to an extended reality device, the method includes: In response to a start-up operation of initializing inertial vision parameters, acquiring inertial information of the extended reality device and capturing a scene image through the extended reality device; Identifying feature point information and scene plane information of the scene image; Determining the inertial vision parameters and the target plane coefficient of the extended reality device according to the plane coefficient to be solved of the scene plane information, the inertial information, and the feature point information, wherein the target plane coefficient is a solution corresponding to the plane coefficient to be solved; The inertial information is acquired based on an inertial measurement unit in the extended reality device, the scene image is acquired by an image acquisition module in the extended reality device, and the inertial vision parameters and target plane coefficients of the extended reality device are determined based on the plane coefficients to be solved of the scene plane information, the inertial information, and the feature point information, including: Determine a three-dimensional coordinate representation corresponding to the feature point information based on the to-be-solved plane coefficients of the scene plane information, wherein the three-dimensional coordinate representation includes a three-dimensional representation of the feature point information in the coordinate system of the image acquisition module; Determining a three-dimensional coordinate conversion relationship and a three-dimensional coordinate projection relationship, wherein the three-dimensional coordinate conversion relationship includes a conversion relationship between a first coordinate system and a second coordinate system at the same time, and the three-dimensional coordinate projection relationship includes a conversion relationship between the first coordinate system and the second coordinate system at different times, wherein the first coordinate system corresponds to the inertial measurement unit, and the second coordinate system corresponds to the image acquisition module; Constructing a target model to be solved according to the three-dimensional coordinate representation, the three-dimensional coordinate transformation relationship and the three-dimensional coordinate projection relationship; The target model to be solved is solved according to the inertial information and the feature point information, and the inertial visual parameters and the target plane coefficient of the extended reality device are determined.

2. The device initialization method according to claim 1, characterized in that: The determining of the three-dimensional coordinate representation corresponding to the feature point information according to the plane coefficient to be solved of the scene plane information includes: Determining a spatial plane expression model corresponding to the scene plane information, where the spatial plane expression model is constructed based on the plane coefficients to be solved for the scene plane information; A three-dimensional coordinate representation is determined based on the image coordinate representation of the feature point information corresponding to the scene plane information, the camera internal parameters corresponding to the extended reality device, and the spatial plane expression model.

3. The device initialization method according to claim 2, characterized in that: The determining of the three-dimensional coordinate representation according to the image coordinate representation of the feature point information corresponding to the scene plane information, the camera intrinsic parameters corresponding to the extended reality device, and the spatial plane expression model includes: Converting the image coordinate representation of the feature point information corresponding to the scene plane information into a normalized coordinate representation according to the intrinsic camera parameters corresponding to the extended reality device; The three-dimensional coordinate representation is determined according to the conversion relationship between the normalized coordinate representation and the three-dimensional coordinate representation and the spatial plane expression model.

4. The device initialization method according to claim 1, wherein: The constructing of the target model to be solved according to the three-dimensional coordinate representation, the three-dimensional coordinate transformation relationship and the three-dimensional coordinate projection relationship includes: Determining a first three-dimensional representation from the three-dimensional coordinate representation, where the first three-dimensional representation is a three-dimensional representation of the feature point information at the initial moment of the start-up operation in a second coordinate system at the initial moment of the image acquisition module; determining a second three-dimensional representation based on the first three-dimensional representation and the three-dimensional coordinate conversion relationship, wherein the second three-dimensional representation represents a three-dimensional representation of a feature point at an initial moment of the startup operation in the first coordinate system of the inertial measurement unit; A target model to be solved is constructed according to the second three-dimensional representation and the three-dimensional coordinate projection relationship.

5. The device initialization method according to claim 4, characterized in that: Determining the three-dimensional coordinate projection relationship includes: Determining an inertial vision observation model corresponding to the inertial measurement unit; Performing pre-integration processing on the inertial visual observation model to obtain a pre-integration representation of inertial parameters; Determining a three-dimensional coordinate transformation relationship for the inertial measurement unit according to the inertial parameter pre-integration representation, the three-dimensional coordinate transformation relationship including a transformation relationship of the coordinate system of the feature point information at different times; A three-dimensional coordinate projection relationship is constructed according to the three-dimensional coordinate transformation relationship and the external parameters between the inertial measurement unit and the image acquisition module.

6. The device initialization method according to claim 5, characterized in that: Solving the target model to be solved according to the inertial information and the feature point information, and determining the inertial visual parameters and target plane coefficients of the extended reality device, includes: Transforming the target model to be solved according to the coupling relationship to obtain a simplified model, wherein the coupling relationship includes a coupling relationship between the plane coefficient to be solved and the inertial information to be solved; The simplified model is solved according to the inertial information and the feature point information to obtain the inertial vision parameters of the inertial measurement unit at the initial moment of the startup operation and the target plane coefficient of the scene image at the initial moment.

7. The device initialization method according to claim 1, characterized in that: The inertial vision parameters include initial inertial vision parameters at the initial moment of the startup operation, and the target plane coefficients include initial plane coefficients of scene plane information at the initial moment; After determining the inertial visual parameters and the target plane coefficient of the extended reality device according to the to-be-solved plane coefficient of the scene plane information, the inertial information, and the feature point information, the method further includes: Determine the three-dimensional coordinates of the feature point information at the initial moment according to the initial plane coefficients; Solving a nonlinear optimization model according to the three-dimensional coordinates, the initial inertial vision parameters, the inertial information, and the feature point information to obtain optimized plane coefficients corresponding to the scene plane information at the initial moment and the optimized inertial vision parameters at the initial moment; The nonlinear optimization model is constructed based on a spatial plane expression model corresponding to a three-dimensional coordinate projection relationship, a pre-integrated representation of inertia parameters, and scene plane information at an initial moment.

8. A device initialization device, characterized in that: Applied to an extended reality device, the apparatus comprises: a response module, configured to obtain inertial information of the extended reality device and capture a scene image through the extended reality device in response to a startup operation of initializing inertial vision parameters; A recognition module, configured to recognize feature point information and scene plane information of the scene image; a determination module, configured to determine the inertial visual parameters and the target plane coefficient of the extended reality device based on the plane coefficient to be solved of the scene plane information, the inertial information, and the feature point information, wherein the target plane coefficient is a solution corresponding to the plane coefficient to be solved; The inertial information is acquired based on an inertial measurement unit in the extended reality device, the scene image is acquired by an image acquisition module in the extended reality device, and the inertial vision parameters and target plane coefficients of the extended reality device are determined based on the plane coefficients to be solved of the scene plane information, the inertial information, and the feature point information, including: Determine a three-dimensional coordinate representation corresponding to the feature point information based on the to-be-solved plane coefficients of the scene plane information, wherein the three-dimensional coordinate representation includes a three-dimensional representation of the feature point information in the coordinate system of the image acquisition module; Determining a three-dimensional coordinate conversion relationship and a three-dimensional coordinate projection relationship, wherein the three-dimensional coordinate conversion relationship includes a conversion relationship between a first coordinate system and a second coordinate system at the same time, and the three-dimensional coordinate projection relationship includes a conversion relationship between the first coordinate system and the second coordinate system at different times, wherein the first coordinate system corresponds to the inertial measurement unit, and the second coordinate system corresponds to the image acquisition module; Constructing a target model to be solved according to the three-dimensional coordinate representation, the three-dimensional coordinate transformation relationship and the three-dimensional coordinate projection relationship; The target model to be solved is solved according to the inertial information and the feature point information, and the inertial visual parameters and the target plane coefficient of the extended reality device are determined.

9. An electronic device, characterized in that: The device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the device initialization method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the device initialization method according to any one of claims 1 to 7 are implemented.

Images