Vision, inertial and active cursor based cooperative positioning method, device and medium

Abstract

The application discloses a vision, inertia and active cursor based cooperative positioning method, device and medium, and relates to the technical field of cooperative positioning of virtual reality and augmented reality. The method comprises the following steps: estimating first attitude information of a first terminal device in a local coordinate system in real time; capturing active light-emitting identifiers of remaining terminal devices in a field of view, and identifying identities of observed terminal devices; calculating relative attitude information between the first terminal device and the observed terminal devices according to pixel positions of the captured active light-emitting identifiers in environment images; exchanging the first attitude information, the identities of the observed terminal devices and the corresponding relative attitude information; and constructing an optimization model of state variables of the first terminal device and the remaining terminal devices. The second attitude information of all terminal devices is obtained by solving the optimization model. The application realizes truly mobile multi-user cooperative positioning of wearing and using, and greatly improves the flexibility and scene adaptability of system deployment.

Description

Technical Field

[0001] This application relates to the field of collaborative positioning technology of virtual reality and augmented reality, and in particular to collaborative positioning methods, devices and media based on vision, inertia and active cursors. Background Technology

[0002] With the rise of social virtual reality and multi-user collaborative augmented reality applications, achieving high-precision and stable relative and absolute positioning among multiple user devices in the same physical space has become a key technical requirement. Currently, mainstream solutions are mainly divided into three categories: systems based on external base stations, which offer high accuracy but are complex to deploy, lack flexibility, and have limited coverage; inside-out solutions based on pure visual inertial odometry, which do not require external facilities, but the independent positioning of each device leads to the inability to automatically align the coordinate systems and suffers from cumulative error drift; and purely wireless positioning solutions, which can provide relative position but lack accuracy and attitude dimensions, making it difficult to meet the needs of immersive interaction.

[0003] However, existing technologies all have significant limitations. Solutions relying on external base stations sacrifice mobility and scene adaptability; while solutions without external facilities, such as pure visual inertial odometry, lack effective geometric constraints to achieve automatic coordinate system alignment and long-term anti-drift in multi-device collaborative scenarios. Although there are technologies that use active markers for relative observation, these technologies are mostly used for pose tracking of master-slave devices and fail to solve the fundamental problem of multiple peer devices forming a network through mutual observation and using this network constraint to achieve globally consistent, high-precision, and drift-resistant collaborative positioning.

[0004] Based on the above analysis, the problems and shortcomings of the existing technology are as follows: In the absence of external infrastructure, existing technologies for multiple virtual reality and augmented reality devices struggle to achieve high-precision, automatic coordinate system alignment and long-term suppression of cumulative drift in collaborative positioning. Summary of the Invention

[0005] This application provides a collaborative positioning method, device, and medium based on vision, inertia, and active cursors, which can solve the problem in the prior art that multiple virtual reality and augmented reality devices are difficult to achieve high-precision, automatic coordinate system alignment, and long-term suppression of cumulative drift in collaborative positioning without external infrastructure.

[0006] In a first aspect, embodiments of this application provide a collaborative localization method based on vision, inertia, and active cursors, applied to a system containing at least two collaborative localization terminal devices. Each terminal device includes a camera, an inertial measurement unit, an active luminous marker, and a wireless communication unit. The method includes: acquiring environmental images and inertial data; estimating the first attitude information of the first terminal device in the local coordinate system in real time; capturing the active luminous markers of the remaining terminal devices within the field of view, and identifying the identity of the observed terminal device based on the preset spatial pattern information of the active luminous markers; calculating the relative attitude information between the first terminal device and the observed terminal devices based on the pixel positions of the captured active luminous markers in the environmental images; exchanging the first attitude information, the identity of the observed terminal devices, and the corresponding relative attitude information through the wireless communication unit to construct an optimization model of the state variables of the first terminal device and the remaining terminal devices; and solving the optimization model to obtain the second attitude information of all terminal devices.

[0007] In one implementation of this application, actively emitting markers on the remaining terminal devices within the field of view are captured, and the identity of the observed terminal devices is identified based on the preset spatial pattern information of the actively emitting markers. Specifically, this includes: synchronously acquiring long-exposure images and short-exposure images at preset intervals; inputting the long-exposure images into a visual inertial odometry system for tracking and matching visual feature points; and inputting the short-exposure images into a marker recognition module to detect and identify the actively emitting markers of the remaining terminal devices through image segmentation and feature extraction algorithms.

[0008] In one implementation of this application, the relative pose information between the first terminal device and the observed terminal device is calculated based on the pixel position of the captured active luminous marker in the image. Specifically, this includes: obtaining the three-dimensional coordinates of the spatial structure of the active luminous marker corresponding to the terminal device based on the identified identity of the observed terminal device; establishing at least four non-coplanar matching point pairs between the three-dimensional coordinates of the spatial structure and the corresponding two-dimensional pixels in the short exposure image; and using a perspective positioning algorithm, based on the matching point pairs, solving for the three-dimensional rotation matrix and translation vector of the observed terminal device relative to the first terminal device to form the actual value of the relative pose.

[0009] In one implementation of this application, a wireless communication unit exchanges first attitude information, the identity of the observed terminal device, and the corresponding relative attitude information to construct an optimization model of the attitude state variables of the first terminal device and the remaining terminal devices. Specifically, the optimization model includes a global optimization model and a local optimization model. When constructing the global optimization model, based on the first attitude information and the identity of the observed terminal devices, the position, attitude, velocity, and IMU deviation of each terminal device at different times are defined as state variables to be optimized. IMU pre-integration constraint edges are constructed between state variables at adjacent times. Visual reprojection constraint edges are constructed based on the tracking and matching of visual feature points. Relative pose constraint edges are constructed based on the relative attitude information. The IMU pre-integration constraint edges, visual reprojection constraint edges, and relative pose constraint edges are jointly incorporated into the factor graph model, and the optimization objective is defined as minimizing the sum of squared errors of the IMU pre-integration constraint edges, visual reprojection constraint edges, and relative pose constraint edges.

[0010] In one implementation of this application, relative pose constraint edges are constructed based on relative pose information. Specifically, this includes: predicting the relative pose estimate between the two terminal devices at a given moment based on the estimated current state variables of the two associated terminal devices in the global optimization model; subtracting the actual relative pose value from the estimated relative pose value and converting it to the Lie algebra space to obtain the error vector of the relative pose constraint edges.

[0011] In one implementation of this application, the method further includes: adding a state variable representing clock deviation to terminal devices that observe each other; and using the state variable representing clock deviation to compensate and correct the timestamp of environmental image acquisition.

[0012] In one implementation of this application, the method further includes: maintaining a local optimization model locally, the local optimization model including its own state and a subset of the states of mutually observed terminal devices; independently solving the local local optimization model to obtain local optimization results; exchanging optimization results through a wireless communication unit, and using a consensus algorithm to coordinate and correct the local optimization results.

[0013] In one implementation of this application, before capturing the active luminous markers of the remaining terminal devices within the field of view, the method further includes: configuring a light-emitting diode array with a unique spatial arrangement pattern for each terminal device and pre-storing three-dimensional coordinate information; and matching the active luminous markers with the light-emitting diode array when identifying the active luminous markers.

[0014] Secondly, embodiments of this application also provide a cooperative localization device based on vision, inertia, and active cursors. The device includes at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform any step of the cooperative localization method based on vision, inertia, and active cursors.

[0015] Thirdly, embodiments of this application also provide a non-volatile computer storage medium for collaborative localization based on vision, inertia, and active cursors, storing computer-executable instructions, which are configured to execute the steps of any one of the collaborative localization methods based on vision, inertia, and active cursors.

[0016] The collaborative positioning, devices, and media based on vision, inertia, and active cursors provided in this application achieve true mobility and plug-and-play functionality, greatly improving deployment flexibility and scenario adaptability. It possesses high precision and strong anti-drift capabilities, effectively suppressing the cumulative errors of independent VIO systems of each device by using visual relative observations with absolute geometric significance between devices as anchor constraints, achieving long-term stable high-precision six-degree-of-freedom relative positioning. It utilizes unique active identifiers to achieve automatic and unambiguous visual recognition of device identity, avoiding complex data associations. Even if a single device is briefly occluded, the system can predict through inertial data and quickly correct after re-observation. It has good scalability and distributed characteristics; new devices only need to be observed by the existing network to integrate, making system expansion simple. It is cost-effective, mainly reusing existing sensors on the devices, and only adding low-cost luminous markers can achieve collaborative positioning effects comparable to external base station systems. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating the collaborative localization method based on vision, inertia, and active cursor provided in the embodiments of this application; Figure 2 A schematic diagram of the overall architecture of the collaborative localization method based on vision, inertia and active cursor provided in the embodiments of this application; Figure 3 A schematic diagram of the internal structure of a collaborative positioning device based on vision, inertia, and active cursor provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] This application provides a collaborative positioning method, device, and medium based on vision, inertia, and active cursor, which solves the problem in the prior art that multiple virtual reality and augmented reality devices are difficult to achieve high-precision, automatic coordinate system alignment, and long-term suppression of cumulative drift in collaborative positioning without external infrastructure.

[0020] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0021] Figure 1 A flowchart illustrating the collaborative localization method based on vision, inertia, and active cursor provided in embodiments of this application. Figure 1 As shown, the collaborative localization method based on vision, inertia, and active cursor provided in this application embodiment is applied to a system containing at least two collaborative localization terminal devices, such as... Figure 2 As shown, each terminal device includes a camera, an inertial measurement unit, an active luminous tag, and a wireless communication unit. The method includes: Step 10: Collect environmental images and inertial data, and estimate the first attitude information of the first terminal device in the local coordinate system in real time.

[0022] In this step, each terminal device independently starts its local visual-inertial odometry system. The visual-inertial odometry system synchronously acquires images of the surrounding environment captured by its own camera, as well as inertial data such as angular velocity and acceleration measured by its own IMU (Inertial Measurement Unit). By extracting and tracking visual feature points from the images, and combining the IMU data for time integration and fusion optimization, each terminal device can calculate and output its position and orientation in real time in a local coordinate system established and maintained by the local VIO (Visual-Inertial Odometry) system. The output information is defined as the first attitude information, which includes translation and rotation in three-dimensional space, that is, six degrees of freedom pose.

[0023] Step 20: Capture the active luminous markers of the remaining terminal devices within the field of view, and identify the identity of the observed terminal devices based on the preset spatial pattern information of the active luminous markers; In this step, each terminal device uses its camera to actively search for active light-emitting markers mounted on other terminal devices in its field of vision. These markers consist of multiple light-emitting units arranged in a specific, predefined spatial geometric pattern.

[0024] As an optional embodiment, before capturing the active luminous markers of the remaining terminal devices within the field of view, the method may further include: configuring a light-emitting diode array with a unique spatial arrangement pattern for each terminal device and pre-storing three-dimensional coordinate information; and matching the active luminous marker with the light-emitting diode array when identifying the active luminous marker.

[0025] In this step, the identification pattern and three-dimensional coordinate model of each device are known in advance and can be shared within the network via wireless communication to form a database that corresponds to device identity and identification model.

[0026] As an optional embodiment, the active luminous markers on the remaining terminal devices within the field of view are captured, and the identity of the observed terminal device is identified based on the preset spatial pattern information of the active luminous markers. Specifically, this may include: Step 201: Simultaneously acquiring long-exposure images and short-exposure images at a preset cycle; Step 202: Inputting the long-exposure image into a visual inertial odometry system for tracking and matching visual feature points; Step 203: Inputting the short-exposure image into a marker recognition module, and detecting and identifying the active luminous markers of the remaining terminal devices through image segmentation and feature extraction algorithms.

[0027] In this step, long-exposure and short-exposure images are acquired simultaneously to resolve the contradiction that a single exposure time cannot simultaneously satisfy both self-motion estimation and long-distance tag recognition. Long-exposure images help to obtain richer environmental texture information in environments with moderate or low lighting. Although slight motion blur may exist, it is still sufficient to support the VIO system's continuous estimation of its own motion. Short exposures can freeze fast-moving or bright light spots, forming clear and sharp point images, which greatly facilitates the accurate extraction of pixel coordinates of light-emitting points through methods such as threshold segmentation and connected component analysis. Subsequently, the spatial relationships of the extracted point set, including relative distances and arrangement shapes, are matched with a pre-stored tag pattern database of all terminal devices to uniquely determine which terminal device is currently being observed, thereby completing the identification process.

[0028] Cooperative processing mechanism for long and short exposure images:

[0029] in, The image intensity values ​​at pixel location u for short and long exposures. Long-exposure images are used for VIO feature tracking to ensure that motion blur is minimized; short-exposure images are used for LED recognition, and the signal-to-noise ratio is enhanced through adaptive threshold segmentation. This represents the radiance of the scene at pixel u and time τ. This represents the noise in the corresponding image.

[0030] Step 30: Based on the pixel position of the captured active luminous marker in the environmental image, calculate the relative attitude information between the first terminal device and the observed terminal device; As an optional embodiment, the relative attitude information between the first terminal device and the observed terminal device is calculated based on the pixel position of the captured active luminous marker in the image, which may specifically include: Step 301: Based on the identified identity of the observed terminal device, obtain the three-dimensional coordinates of the spatial structure of the active luminous identifier corresponding to the terminal device.

[0031] In this step, each terminal device is equipped with a unique active light-emitting label, which is a specially arranged LED array. The three-dimensional coordinates of each light-emitting element in the label in the device's own coordinate system have been calibrated before the system leaves the factory or during the initialization stage and are pre-stored in the device's local memory.

[0032] Step 302: Establish at least four non-coplanar matching point pairs between the three-dimensional coordinates of the spatial structure and the corresponding two-dimensional pixels of the short exposure image.

[0033] In this step, the coordinates of two-dimensional pixels belonging to the same group of light-emitting points detected and identified from the short-exposure image are precisely correlated. Each correct correlation constitutes a matching pair of two-dimensional points in the image and three-dimensional points in space. In order to achieve stable and unique pose calculation, at least four such matching point pairs need to be established, and these three-dimensional points should not all be coplanar in space, so as to provide sufficient geometric constraints to solve for the six unknown parameters: three rotation angles and three translations.

[0034] Step 303: Using the perspective positioning algorithm, based on the matching point pairs, the three-dimensional rotation matrix and translation vector of the observed terminal device relative to the first terminal device are obtained, forming the actual value of the relative pose.

[0035] In this step, the optimal rigid body transformation is solved so that when the 3D points of the device identifier are projected onto the camera imaging plane of the device through this transformation, the reprojection error between the projected points and the 2D points actually observed in the image is minimized.

[0036] Specifically, the PnP (Perspective-n-Point) algorithm can be used to solve for the device from the 2D-3D correspondence of at least 4 LED points. Relative to equipment pose of camera coordinate system ,set up LED dots in equipment Coordinates in the camera coordinate system:

[0037] The PnP problem can be represented as solving a rotation matrix. Translation vector Make:

[0038] π( ) represents the projection model of the camera, including intrinsic parameters. This optimization problem can be solved by EPnP, UPnP or nonlinear optimization methods; R and t are the rotation matrix and translation vector to be found, which are the poses of device n relative to the camera coordinate system of device m. This represents the total number of LED dots; To project 3D points onto the image plane based on the currently estimated pose; The actual observed pixel coordinates.

[0039] Step 40: Exchange the first attitude information, the identity of the observed terminal device and the corresponding relative attitude information through the wireless communication unit, and construct an optimization model of the state variables of the first terminal device and the remaining terminal devices.

[0040] In this step, each terminal device shares its own sensed data, its own state, and relative observations through a wireless network, and uses this data to jointly construct a mathematical model that can simultaneously optimize the pose of all devices.

[0041] As an optional embodiment, the first attitude information, the identity of the observed terminal device and the corresponding relative attitude information are exchanged through the wireless communication unit to construct an optimization model of the attitude state variables of the first terminal device and the remaining terminal devices. Specifically, it may include: Step 401: The optimization model includes a global optimization model and a local optimization model.

[0042] In this step, all data is aggregated into a central node, which can be any device, to construct and solve a unified optimization problem that includes the complete state of all devices in the network, and directly outputs the global optimal solution. To improve scalability and robustness, each device can maintain a local optimization model locally, which only includes its own state and a subset of the states of neighboring devices that have direct communication or observation relationships with it. Each device solves its local model in parallel and negotiates with each other to make all local solutions converge to global consistency.

[0043] Step 402: When constructing the global optimization model, based on the first attitude information and the identity of the observed terminal device, the position, attitude, velocity and IMU deviation of each terminal device at different times are defined as state variables to be optimized.

[0044] In this step, in order to perform optimization, we first need to define the optimization variables, the first... Each device in time The state variables are:

[0045] in: Indicates the device's position in the global coordinate system. It is a quaternion representation of the device attitude. This indicates the device's velocity in the global coordinate system. Indicates gyroscope deviation. This indicates the accelerometer deviation.

[0046] Step 403: Construct IMU pre-integration constraint edges between state variables at adjacent time points.

[0047] In this step, the IMU measurement includes both true values ​​and noise:

[0048] in, This represents the gravity vector in the global coordinate system. ; The gyroscope measurement at time t (including noise and zero bias); The actual angular velocity at time t; The gyroscope has zero bias at time t; Gyroscope noise measurement; The accelerometer measurement is at time t; This is the transpose of the rotation matrix from the global coordinate system G to the body coordinate system; The actual acceleration at time t; The accelerometer bias is zero at time t; Accelerometer measurement noise.

[0049] equipment Observed equipment LED signage, equipment The Each LED The coordinates in the coordinate system are The coordinates in the device's body coordinate system are:

[0050] in, The coordinates of the j-th LED of device n in its own LED coordinate system Ln; The coordinates of the j-th LED of device n in its body coordinate system Bn; Rotation matrix from LED coordinate system to body coordinate system; Translation vector from the LED coordinate system to the body coordinate system.

[0051] The coordinates in the global coordinate system are:

[0052] in, , The orientation of device n at time i (rotation from the device itself to the global plane); .

[0053] The LED in the device The projection on the camera image is:

[0054] in, Device m observes the pixel coordinates (including noise) of the j-th LED of device n. To observe noise; The rotation matrix from the camera to the body of device m; The inverse of the attitude of device m at time i; The position of device m in the global coordinate system G at time i; The translation vector from the camera of device m to the body.

[0055] Equations of motion in continuous time:

[0056] in Indicates the antisymmetric matrix operator, This is the derivative of position with respect to time (i.e., velocity). This is the derivative of velocity with respect to time (i.e., acceleration). The time derivative of the rotation matrix.

[0057] In this step, in two image frames and Between these points, the IMU pre-integral quantity is:

[0058] in, This represents the exponential mapping from Lie algebras to Lie groups. The relative rotational change from time i to j; The change in relative velocity from time i to j; Δt represents the IMU sampling time interval, and the index k indicates the IMU sampling time from i to j-1. Step 404: Construct visual reprojection constraint edges based on the tracking and matching of visual feature points.

[0059] In this step, for the first VIO system of each device, visual feature points The coordinates in the global coordinate system are The observation error of its projection onto the image plane is:

[0060] in, Let be the visual reprojection error of the k-th device. Let i be the state variable of the k-th device at time i. The 3D coordinates of the l-th visual feature point in the global coordinate system G Observed pixel coordinates (2D) on the image. For camera projection function, For image observations, and Let the rotation matrix and translation vector be the distance from the camera to the device body. Let G be the rotation matrix from the global coordinate system G to the device body coordinate system k. The position of device k in the global coordinate system G at time i.

[0061] Step 405: Construct relative pose constraint edges based on relative pose information; As an optional embodiment, relative pose constraint edges are constructed based on relative pose information, which may specifically include: Step 4051: Based on the current state variable estimates of two associated terminal devices in the global optimization model, the relative pose estimate between the two terminal devices at this moment is predicted.

[0062] In this step, during the optimization iteration process, the pose of device n relative to the camera of device m can be predicted based on the current state estimates of device m and device n at time i, as well as the extrinsic parameters from the camera of device m to the body.

[0063] Step 4052: Subtract the actual relative pose value from the estimated relative pose value and transform it to the Lie algebra space to obtain the error vector of the relative pose constraint edge.

[0064] In this step, the actual observed values ​​obtained through PnP calculation are compared with the predicted values ​​from the previous step to construct the error. This error has two commonly used definitions: Define the error directly on the pose manifold:

[0065] in Describe the logarithmic mapping from Lie groups to Lie algebras. The IMU pre-integrated rotation from time i to i+1; The change in IMU pre-integration velocity from time i to i+1; Change in IMU pre-integration position from time i to i+1.

[0066] Inter-equipment relative observation error term:

[0067] in The relative pose estimate obtained through PnP (from device n body to device m camera). For equipment The extrinsic parameter matrix from the camera to the body. The pose of device m at time i (from the device itself to the global position). The inverse of the pose of device n at time i (from global to the subject).

[0068] Alternatively, the reprojection error of the LED points can be used directly, comparing the observed LED pixel coordinates with the projected coordinates predicted based on the device status estimate:

[0069] Step 406: Incorporate the IMU pre-integration constraint edge, the visual reprojection constraint edge, and the relative pose constraint edge into the factor graph model, and define the optimization objective as minimizing the sum of squared errors of the IMU pre-integration constraint edge, the visual reprojection constraint edge, and the relative pose constraint edge.

[0070] In this step, all the aforementioned state variables are combined as nodes of the graph, and various constraint edges are combined as factors connecting the nodes to form a factor graph. Constructing a factor graph that includes all device states and constraints transforms the entire cooperative localization problem into a large-scale nonlinear least squares optimization problem, which can be expressed as:

[0071] in, The optimal set of state variables, Let k be the set of all state variables for all devices, where k is the total number of devices. For equipment A collection of state-time indexes. For equipment The observed set of visual features For a set of devices that observe each other, For equipment Observed equipment A collection of time indices.

[0072] By solving this optimization problem, a global optimal state estimate that minimizes the overall error of all self-motion constraints, environmental observation constraints, and relative observation constraints between devices is obtained.

[0073] Step 50: Solve the optimization model to obtain the second pose information of all terminal devices.

[0074] In this step, the optimization problem can be expressed as a non-linear least squares problem:

[0075] Where is the stacked vector of all error terms (IMU pre-integration error, visual reprojection error, relative pose error between devices), is the corresponding covariance matrix block diagonal matrix, which is used to reasonably balance the confidence of different sensors and observations.

[0076] The Gauss-Newton method or Levenberg-Marquardt algorithm is used for iterative solution:

[0077] Where, is the Jacobian matrix, is the damping factor, which is used to control the iteration step size and ensure convergence. By iteratively updating the state variables until convergence (the error change or update amount is less than the set threshold), Σ is the covariance matrix block diagonal matrix of all observation noises, I is the identity matrix, is the increment of the state variable (the update amount to be solved), is the stacked vector of all error terms.

[0078] When the optimization solver converges, the set of optimal state estimates X contains the position and orientation of each terminal device in the global coordinate system G, which is the final output result and is called the second pose information. Compared with the first pose information estimated independently by the device in step 10, which may have cumulative drift, the second pose information significantly suppresses drift by tightly integrating the absolute geometric observation constraints between devices, achieving high-precision and consistent positioning of multiple devices in a unified global coordinate system, providing a reliable spatial sharing basis for upper-layer VR / AR applications.

[0079] As an optional embodiment, the method may further include: adding state variables representing clock offsets for terminal devices with mutual observations; using the state variables of clock offsets to compensate and correct the timestamps of environmental image acquisition.

[0080] In this step, due to the clock non-synchronization between devices, it is necessary to align the observation data in time. Let device and device The clock deviation is The observation equation is then corrected to:

[0081] Clock skew between device m and device n It can be included in the state vector as an optimization variable; The actual time after clock skew compensation.

[0082] As an optional embodiment, the method may further include: maintaining a local optimization model locally, the local optimization model including its own state and a subset of the states of mutually observed terminal devices; independently solving the local optimization model to obtain local optimization results; exchanging optimization results through a wireless communication unit, and using a consensus algorithm to coordinate and correct the local optimization results.

[0083] In this step, each device maintains a local factor graph and performs partial optimization, achieving global consistency through consistency constraints. Local optimization problem:

[0084] in, The set of local state variables maintained by device k. For consistency penalty function, To be compatible with equipment A collection of devices that are related by observation.

[0085] Through repeated local computation and information exchange between neighbors, the local estimates of all devices will gradually converge, eventually converging to a result that is basically consistent with the centralized global optimization. Under this architecture, when a new device joins the system, it only needs to establish observation and communication with some devices in the existing network to quickly integrate into the cooperative positioning system, achieving good scalability.

[0086] In summary, the workflow of this application is as follows: Each device starts its local VIO system and constructs an independent local map; device IDs and LED identification information are exchanged via wireless communication; communication connections and time synchronization are established between devices; cameras acquire double-exposure image sequences; IMUs continuously acquire inertial data; long-exposure images are sent to the local VIO engine for self-pose estimation; short-exposure images are sent to the light tag recognition engine to detect and identify the LEDs of other devices; each device periodically broadcasts data packets containing its own status, IMU data, and LED models; receives and parses data packets from other devices; inputs its own VIO output, LED observation results, and received data from other devices into the collaborative positioning and fusion module; constructs and solves a global optimization problem to obtain the precise poses of all devices in a unified coordinate system; and outputs the optimized poses to the upper-layer application. Furthermore, data collection, processing, and optimization continue; when new devices are added, they are automatically incorporated into the optimization framework; when devices leave, the system automatically adjusts the optimization structure.

[0087] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a collaborative positioning device based on vision, inertia, and active cursors, the structure of which is as follows: Figure 3 As shown.

[0088] Figure 3 This is a schematic diagram of the internal structure of a collaborative positioning device based on vision, inertia, and active cursor, provided as an embodiment of this application. Figure 3 As shown, the device includes: At least one processor 301; And a memory 302 that is communicatively connected to at least one processor; The memory 302 stores instructions that can be executed by at least one processor, which are executed by at least one processor 301 to enable at least one processor 301 to: perform any step of a cooperative localization method based on vision, inertia and active cursor.

[0089] Some embodiments of this application provide corresponding to Figure 1 A non-volatile computer storage medium for co-localization based on vision, inertia, and active cursor, storing computer-executable instructions, which are configured to execute the steps of any one of the co-localization methods based on vision, inertia, and active cursor.

[0090] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0091] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0092] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0097] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0098] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0099] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0100] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A collaborative localization method based on vision, inertial measurement unit, and active cursor, applied to a system comprising at least two collaborative localization terminal devices, each terminal device including a camera, an inertial measurement unit, an active luminous marker, and a wireless communication unit, characterized in that, The method includes: Collect environmental images and inertial data, and estimate the first attitude information of the first terminal device in the local coordinate system in real time; The active luminous markers of the remaining terminal devices within the field of view are captured, and the identity of the observed terminal device is identified based on the preset spatial pattern information of the active luminous markers. Based on the captured pixel position of the active luminous marker in the environmental image, the relative attitude information between the first terminal device and the observed terminal device is calculated. The first attitude information, the identity of the observed terminal device, and the corresponding relative attitude information are exchanged through the wireless communication unit to construct an optimization model of the state variables of the first terminal device and the remaining terminal devices. Solve the optimization model to obtain the second attitude information of all terminal devices.

2. The collaborative localization method based on vision, inertia, and active cursor as described in claim 1, characterized in that, The process of capturing the active luminous markers on the remaining terminal devices within the field of view, and identifying the identity of the observed terminal devices based on the preset spatial pattern information of the active luminous markers, specifically includes: Simultaneously acquire long-exposure and short-exposure images according to a preset cycle; The long-exposure image is input into a visual inertial odometry system for tracking and matching visual feature points; The short-exposure image is input into the identifier recognition module, and the active luminous identifiers of the remaining terminal devices are detected and identified through image segmentation and feature extraction algorithms.

3. The collaborative localization method based on vision, inertia, and active cursor as described in claim 2, characterized in that, The step of calculating the relative attitude information between the first terminal device and the observed terminal device based on the captured pixel position of the actively emitting marker in the image specifically includes: Based on the identified identity of the observed terminal device, obtain the three-dimensional spatial coordinates of the active luminous identifier corresponding to the terminal device; Establish at least four non-coplanar matching point pairs between the three-dimensional coordinates of the spatial structure and the corresponding two-dimensional pixels of the short-exposure image; Using the perspective positioning algorithm, based on the matching point pairs, the three-dimensional rotation matrix and translation vector of the observed terminal device relative to the first terminal device are obtained, thus forming the actual value of the relative pose.

4. The collaborative localization method based on vision, inertia, and active cursor as described in claim 3, characterized in that, The step of exchanging the first attitude information, the identity of the observed terminal device, and the corresponding relative attitude information through the wireless communication unit to construct an optimization model of the attitude state variables of the first terminal device and the remaining terminal devices specifically includes: The optimization model includes a global optimization model and a local optimization model; When constructing the global optimization model, based on the first attitude information and the identity of the observed terminal device, the position, attitude, velocity and IMU deviation of each terminal device at different times are defined as state variables to be optimized. Construct IMU pre-integration constraint edges between the state variables at adjacent time points; Based on the tracking and matching of the visual feature points, visual reprojection constraint edges are constructed; Based on the relative pose information, construct relative pose constraint edges; The IMU pre-integration constraint edge, visual reprojection constraint edge, and relative pose constraint edge are all incorporated into the factor graph model, and the optimization objective is defined as minimizing the sum of squared errors of the IMU pre-integration constraint edge, visual reprojection constraint edge, and relative pose constraint edge.

5. The collaborative localization method based on vision, inertia, and active cursor as described in claim 4, characterized in that, The step of constructing relative pose constraint edges based on the relative pose information specifically includes: Based on the estimated values ​​of the current state variables of the two associated terminal devices in the global optimization model, the estimated value of the relative pose between the two terminal devices at this moment is predicted. The difference between the actual relative pose value and the estimated relative pose value is calculated and transformed into Lie algebra space to obtain the error vector of the relative pose constraint edge.

6. The collaborative localization method based on vision, inertia, and active cursor as described in claim 1, characterized in that, The method further includes: Add a state variable to represent clock deviation for terminal devices that observe each other; The timestamps of environmental image acquisition are compensated and corrected using the state variable of the clock deviation.

7. The collaborative localization method based on vision, inertia, and active cursor as described in claim 4, characterized in that, The method further includes: The local optimization model is maintained locally, and the local optimization model includes its own state and a subset of the states of mutually observed terminal devices; Solve the local optimization model independently to obtain the local optimization results; The optimization results are exchanged through the wireless communication unit, and the local optimization results are coordinated and corrected using a consensus algorithm.

8. The collaborative localization method based on vision, inertia, and active cursor as described in claim 1, characterized in that, Before the actively emitting identifier of the remaining terminal devices within the capture field of view, the method further includes: Each terminal device is configured with a unique spatial arrangement pattern of light-emitting diode arrays, and its three-dimensional coordinate information is pre-stored; When identifying the active light-emitting identifier, the active light-emitting identifier is matched with the light-emitting diode array.

9. A collaborative positioning device based on vision, inertia, and active cursor, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Perform the steps of the collaborative localization method based on vision, inertia, and active cursor as described in any one of claims 1-8.

10. A non-volatile computer storage medium based on visual, inertial, and active cursor cooperative positioning, storing computer-executable instructions, characterized in that: The computer-executable instructions are set as follows: Perform the steps of the collaborative localization method based on vision, inertia, and active cursor as described in any one of claims 1-8.

Images