Multi-source positioning data fusion method, device and equipment

Abstract

The invention provides a multi-source positioning data fusion method, device and equipment. The method comprises the following steps: reading ultra-wideband positioning data and visual positioning data in an asynchronous mode; preprocessing the ultra-wideband positioning data to obtain preprocessed ultra-wideband positioning data; detecting the visual positioning data, and when the visual positioning data is detected to reach a motion threshold value, respectively collecting a plurality of ultra-wideband calibration points and visual calibration points in the pre-processed ultra-wideband positioning data and visual positioning data; when the ultra-wideband calibration points and the visual calibration points reach a preset number and the visual positioning data are completely converted into the ultra-wideband coordinate system, fusing the visual data in the ultra-wideband coordinate system with the preprocessed ultra-wideband positioning data to obtain fused positioning data, and applying the fused positioning data to the target AVamp; a VR head display device; the multi-person large space ARamp is improved; and interactive experience between people and people / virtual scenes in a VR content scene is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of large-space AR&VR content production, and particularly relates to a multi-source positioning data fusion method, device and equipment. Background Art

[0002] Large-space AR&VR content refers to various content forms in a large space produced through augmented reality technology (AR) and virtual reality technology (VR). Compared with traditional AR&VR content, which needs to be experienced within a specified safe range (usually between several square meters and more than a dozen square meters), large-space AR&VR content can break the boundaries of the safe range, enabling the experiencer to interact with the virtual scene in a large space of dozens of square meters or even thousands or tens of thousands of square meters.

[0003] The core point of large-space AR&VR content production is the person positioning technology. To achieve the interaction between the experiencer and the virtual space, it is necessary to obtain the motion information of the experiencer and then put the motion information into the virtual space to achieve the interactive effect of the experiencer "moving" in the virtual space.

[0004] Currently, the positioning data used in AR&VR content mainly depends on the head-mounted display device (HMD). In conventional HMDs, the positioning technologies mainly include visual simultaneous localization and mapping (SLAM), visual-inertial measurement unit (IMU) SLAM technology, etc.

[0005] The SLAM technology calculates the position of the device in space by using the point cloud information obtained by the visual sensor, and uses the IMU sensor to assist the visual positioning information. Therefore, the spatial position calculated by this type of technology will use the point where the head-mounted device starts running as the origin of the spatial coordinates, and the direction values of the spatial coordinates will also change according to the different starting points of the device operation. This spatial positioning information does not cause problems when used in single-person small-space AR&VR content; however, in multi-person large-space AR&VR content, the demand for spatial positioning is higher. Each user has a unique spatial coordinate in the actual space (and this coordinate is based on a fixed world coordinate system), and relative position and orientation relationships are generated. This relative relationship needs to be mapped into the virtual space so that the positions and orientations of users in the virtual space are the same as those in the actual space, ultimately serving the interaction between users and the virtual space and between users. Due to the limitations of the SLAM positioning technology, most current AV&VR head-mounted devices cannot obtain accurate position and orientation relationships when experiencing multi-person large-space content, or need to use the large-space scanning solution provided by the head-mounted device manufacturer to scan the point cloud images in the entire large-space using the head-mounted device, and then share this point cloud image through multiple head-mounted devices to achieve the positioning between people and between people and the scene. This makes the interaction between users and the virtual space and between users in the multi-person large-space content scenario a difficult or extremely cumbersome problem to solve. Summary of the Invention

[0006] The present invention provides a multi-source positioning data fusion method, device, and equipment, aiming to improve the interaction experience between people / virtual scenes in the multi-person large-space AR&VR content scenario.

[0007] To achieve the above objective, the present invention provides a multi-source positioning data fusion method, including:

[0008] Step 1, reading ultra-wideband positioning data and visual positioning data in an asynchronous manner;

[0009] Step 2, preprocessing the ultra-wideband positioning data to obtain the preprocessed ultra-wideband positioning data;

[0010] Step 3, detecting the visual positioning data. When it is detected that the visual positioning data reaches the motion threshold, collect a number of ultra-wideband calibration points and visual calibration points from the preprocessed ultra-wideband positioning data and the visual positioning data respectively;

[0011] Step 4, when the number of ultra-wideband calibration points and visual calibration points reaches the preset number, convert the visual positioning data to the ultra-wideband coordinate system and determine whether the conversion is completed;

[0012] If the conversion is completed, the visual data in the ultra-wideband coordinate system is fused with the preprocessed ultra-wideband positioning data to obtain the fused positioning data, and the fused positioning data is applied to the target AV&VR headset device;

[0013] If the conversion is not completed, several ultra-wideband calibration points and visual calibration points are recollected, and step 4 is executed again.

[0014] Furthermore, step 1 includes:

[0015] Continuously read multiple positioning frames sent by the ultra-wideband device through asynchronous serial communication;

[0016] Parse each positioning frame to obtain the key attributes of each positioning frame as ultra-wideband positioning data, and store the ultra-wideband positioning data in the ultra-wideband queue;

[0017] Parse the positioning message sent by the asynchronous UDP remote monitoring system to obtain the three-dimensional coordinates and attitude information as visual positioning data, and store the visual positioning data in the visual queue.

[0018] Furthermore, preprocess the ultra-wideband positioning data to obtain the preprocessed ultra-wideband positioning data, including:

[0019] Input the ultra-wideband positioning data into the UKF filtering algorithm;

[0020] Define the state variables of the UKF filtering algorithm, and construct the prediction equation, observation equation and covariance matrix;

[0021] Generate several sigma points in the ultra-wideband positioning data, and substitute each sigma point into the prediction equation for solution to obtain the prior prediction result;

[0022] Calculate the prior mean and prior covariance based on the prior prediction result;

[0023] Substitute each prior sigma point into the observation equation for calculation to obtain the scaling factor of the measurement space;

[0024] Calculate the measurement prediction mean and measurement prediction variance using the scaling factor of the measurement space;

[0025] Calculate the cross covariance based on the prior prediction result, prior mean, scaling factor of the measurement space, and measurement prediction mean;

[0026] Calculate the Kalman gain using the cross covariance and measurement covariance;

[0027] Calculate the posterior state mean based on the Kalman gain, prior mean, and measurement prediction mean, and use the first two dimensions of the posterior state mean as the preprocessed ultra-wideband positioning data.

[0028] Furthermore, the expression for calculating the posterior state mean based on the Kalman gain, the prior mean, and the measurement prediction mean is as follows:

[0029]

[0030] where, represents the posterior state mean, represents the prior mean, and K k+1 represents the Kalman gain, and z k+1 represents the actual measurement value, represents the measurement prediction mean.

[0031] Furthermore, converting the visual positioning data to the ultra-wideband coordinate system includes:

[0032] Converting both the ultra-wideband calibration points and the visual calibration points to the coordinate system centered on the centroid, obtaining the ultra-wideband calibration points in the coordinate system centered on the centroid and the visual calibration points in the coordinate system centered on the centroid;

[0033] Constructing a cross-covariance matrix using the ultra-wideband calibration points in the coordinate system centered on the centroid and the visual calibration points in the coordinate system centered on the centroid, and after performing singular value decomposition on the cross-covariance matrix, solving for the optimal rotation matrix;

[0034] Calculating the translation vector using the centroid of the ultra-wideband calibration points, the centroid of the visual calibration points, and the optimal rotation matrix;

[0035] Converting the visual positioning data to the ultra-wideband coordinate system based on the translation vector and the optimal rotation matrix.

[0036] Furthermore, the conversion expression for converting the visual positioning data to the ultra-wideband coordinate system based on the translation vector and the optimal rotation matrix is:

[0037]

[0038] where, represents the visual data in the ultra-wideband coordinate system, X vision represents the visual positioning data, T represents the translation vector, and R represents the optimal rotation matrix.

[0039] Furthermore, the calculation expression for fusing the visual data in the ultra-wideband coordinate system with the preprocessed ultra-wideband positioning data is:

[0040]

[0041] where, x fused and y fused respectively represent the x coordinate and the y coordinate of the positioning data after weighted average fusion, and wUWB Denotes the weight of the pre - processed ultra - wideband positioning data, w Vision Denotes the weight of the visual positioning data, x ukf , y ukf Respectively denote the x - coordinate and y - coordinate of the pre - processed ultra - wideband positioning data Respectively denote the x - coordinate and y - coordinate of the visual positioning data in the ultra - wideband coordinate system

[0042] Furthermore, before obtaining the fused positioning data, it also includes:

[0043] Filter the fused positioning data by the UKF filtering algorithm, and use the filtered positioning data as the fused positioning data

[0044] The present invention also provides a multi - source positioning data fusion device, including:

[0045] A reading module, used to read ultra - wideband positioning data and visual positioning data in an asynchronous manner

[0046] A first processing module, used to pre - process the ultra - wideband positioning data to obtain the pre - processed ultra - wideband positioning data

[0047] A second processing module, used to detect the visual positioning data. When it detects that the visual positioning data reaches the motion threshold, collect a number of ultra - wideband calibration points and visual calibration points from the pre - processed ultra - wideband positioning data and visual positioning data respectively

[0048] A conversion module, used to convert the visual positioning data to the ultra - wideband coordinate system and determine whether the conversion is completed when the number of ultra - wideband calibration points and visual calibration points reaches the preset quantity

[0049] If the conversion is completed, fuse the visual data in the ultra - wideband coordinate system with the pre - processed ultra - wideband positioning data to obtain the fused positioning data, and apply the fused positioning data to the target AV&VR headset device

[0050] If the conversion is not completed, re - collect a number of ultra - wideband calibration points and visual calibration points, and return to execute step 4

[0051] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multi - source positioning data fusion method

[0052] The above - mentioned solution of the present invention has the following beneficial effects:

[0053] The present invention reads ultra-wideband positioning data and visual positioning data asynchronously; preprocesses the ultra-wideband positioning data to obtain the preprocessed ultra-wideband positioning data; detects the visual positioning data, and when it is detected that the visual positioning data reaches the motion threshold, a number of ultra-wideband calibration points and visual calibration points are respectively collected from the preprocessed ultra-wideband positioning data and the visual positioning data; when the ultra-wideband calibration points and the visual calibration points reach the preset number and the visual positioning data is completely converted to the ultra-wideband coordinate system, the visual data in the ultra-wideband coordinate system is fused with the preprocessed ultra-wideband positioning data to obtain the fused positioning data, and the fused positioning data is applied to the target AV&VR headset device; compared with the prior art, the present invention preprocesses the ultra-wideband positioning data and collects calibration points, collects calibration points for the visual positioning data, performs coordinate conversion on the visual positioning data based on the centroid of the calibration points and determines whether the conversion is completed. If the conversion is completed, the visual data in the ultra-wideband coordinate system is fused with the preprocessed ultra-wideband positioning data, otherwise a number of ultra-wideband calibration points and visual calibration points are collected again, which can solve the problem that in the visual positioning mode of traditional head-mounted display devices, it is impossible or relatively cumbersome to perform multi-person relative position positioning in a large multi-person space AR&VR content scenario, and improve the interaction experience between people / virtual scenes in a large multi-person space AR&VR content scenario.

[0054] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation part. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a schematic flowchart of an embodiment of the present invention;

[0056] Figure 2 is a detailed flowchart in an embodiment of the present invention;

[0057] Figure 3 is a schematic diagram of the principle of the synchronization processing mechanism in an embodiment of the present invention;

[0058] Figure 4 is a schematic structural diagram of a multi-source positioning data fusion device in an embodiment of the present invention;

[0059] Figure 5 is a schematic structural diagram of a terminal device in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0060] To make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0061] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the accompanying drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be understood as a limitation to the present invention. In addition, the terms "first", "second", "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0062] In the description of the present invention, it should be noted that unless otherwise clearly defined and limited, the terms "installed", "connected", "connected" should be understood in a broad sense. For example, it can be a locking connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0063] In addition, the technical features involved in different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0064] The present invention provides a multi-source positioning data fusion method, device and equipment for existing problems.

[0065] As Figure 1 、 Figure 2 shown, an embodiment of the present invention provides a multi-source positioning data fusion method, including:

[0066] Step 1, reading ultra-wideband positioning data and visual positioning data in an asynchronous manner;

[0067] Step 2, preprocessing the ultra-wideband positioning data to obtain the preprocessed ultra-wideband positioning data;

[0068] Step 3, detecting the visual positioning data. When it is detected that the visual positioning data reaches the motion threshold, a number of ultra-wideband calibration points and visual calibration points are respectively collected from the preprocessed ultra-wideband positioning data and the visual positioning data;

[0069] Step 4, when the number of ultra-wideband calibration points and visual calibration points reaches the preset number, convert the visual positioning data to the ultra-wideband coordinate system and determine whether the conversion is completed;

[0070] If the conversion is completed, fuse the visual data in the ultra-wideband coordinate system with the preprocessed ultra-wideband positioning data to obtain the fused positioning data, and apply the fused positioning data to the target AV&VR headset device;

[0071] If the conversion is not completed, collect several ultra-wideband calibration points and visual calibration points again, and return to execute Step 4.

[0072] Specifically, Step 1 includes:

[0073] Continuously read multiple positioning frames sent by the ultra-wideband device through asynchronous serial communication;

[0074] Parse each positioning frame to obtain the key attributes of each positioning frame as ultra-wideband (UWB, UltraWideband) positioning data, abbreviated as UWB positioning data, and store the ultra-wideband positioning data in the ultra-wideband queue;

[0075] Parse the positioning message sent by the asynchronous UDP remote monitoring system to obtain the three-dimensional coordinates and attitude information as visual positioning data, and store the visual positioning data in the visual queue.

[0076] In the embodiment of the present invention, the key attributes of each positioning frame include the ID, three-dimensional coordinates, distance information, etc. of the node, and these key attributes are encapsulated as ultra-wideband positioning data.

[0077] In the embodiment of the present invention, the positioning message sent by the asynchronous UDP remote monitoring system needs to be parsed in JSON to extract the three-dimensional coordinates and attitude of the target as visual positioning data.

[0078] In the embodiment of the present invention, the ultra-wideband queue and the visual queue are set to store the positioning data collected by different devices, and high-concurrency reading and writing in an asynchronous environment are ensured. If the queue becomes saturated, a discard or rate-limiting mechanism is used to avoid system blocking.

[0079] In order to reduce the jitter error of the ultra-wideband positioning data, the embodiment of the present invention preprocesses the ultra-wideband positioning data to obtain the preprocessed ultra-wideband positioning data, including:

[0080] Input the ultra-wideband positioning data into the UKF filtering algorithm;

[0081] Define the state quantity of the UKF filtering algorithm as where, (x,y) represents the position coordinates on the two-dimensional plane, and represents the velocity components in the x and y directions;

[0082] Through a linear prediction model The prediction equation is constructed as where Δt represents the discrete time step, which is set to 1.0 in the embodiments of the present invention, and x k+1 represents the motion state at the next moment, and x k represents the motion state at the current moment, and k represents the index of the discrete time step;

[0083] Since the embodiments of the present invention only process the position information, only the position is observed, and the observation equation is where z k represents the observed position;

[0084] The constructed covariance matrix includes a process noise covariance matrix a measurement noise covariance matrix an initial covariance matrix where represents the process noise magnitude in the position dimension, represents the process noise magnitude in the velocity dimension, represents the variance of the X and Y position observation dimensions, represents the process noise magnitudes in the position and velocity dimensions at the initial stage;

[0085] Generate a number of sigma points in the ultra-wideband positioning data, and the generation formula is:

[0086]

[0087] The method characterizes the characteristics of the Gaussian distribution under non-linear transformation by selecting a number of Sigma points; represents the state mean (posterior mean) after fusing the measurement at time step k; P k|k represents the state covariance (posterior covariance) obtained after fusing the measurement at time step k; n represents the dimension of the state vector, which is n = 4 in the embodiments of the present invention; λ is a combined coefficient related to the hyperparameters α, β, κ of the Unscented Transform, and its calculation formula is λ = α 2(n+κ) -n, and in the embodiments of the present invention, the three hyperparameters are respectively set to α = 0.1, β = 2, k = 0;

[0088] Substitute each sigma point into the prediction equation for solution to obtain the prior prediction result, and the expression is:

[0089]

[0090] Calculate the prior mean and prior covariance based on the prior prediction results, where:

[0091] The expression for the prior mean is:

[0092]

[0093] where, represents the prior mean, represents the weight of the i-th prior prediction result;

[0094] The expression for the prior covariance is:

[0095]

[0096] where, represents the weight;

[0097] Substitute each prior sigma point into the observation equation for calculation to obtain the scale factor in the measurement space The expression is:

[0098]

[0099] Use the scale factor γ in the measurement space i to calculate the measurement prediction mean and measurement prediction variance, where:

[0100]

[0101] Based on the prior prediction results, prior mean, scale factor γ in the measurement space i and measurement prediction mean, calculate the cross covariance The expression is:

[0102]

[0103] Use the cross covariance and measurement covariance to calculate the Kalman gain K k+1 The expression is:

[0104]

[0105] Based on the Kalman gain, prior mean, and measurement prediction mean, calculate the posterior state mean, and use the first two dimensions of the numbers in the posterior state mean as the preprocessed ultra-wideband positioning data.

[0106] In an embodiment of the present invention, the UKF (Unscented Kalman Filter) filtering algorithm is an unscented Kalman filtering algorithm, which is a filtering algorithm for processing nonlinear systems. By approximating the probability distribution with sigma points, it avoids the problem of calculating the Jacobian matrix in the Extended Kalman Filter (EKF), improving the estimation accuracy. UKF is applicable to nonlinear systems, and its basic principle includes constructing sigma points through the UT transformation, calculating the mean and variance of the nonlinear function, and thus obtaining the posterior probability density of the state.

[0107] Specifically, the expression for calculating the posterior state mean based on the Kalman gain, prior mean, and measurement prediction mean is:

[0108]

[0109] Among them, represents the posterior state mean, represents the prior mean, and K k+1 represents the Kalman gain, z k+1 represents the actual measurement value, represents the measurement prediction mean.

[0110] Most preferably, in an embodiment of the present invention, when receiving visual positioning data, it is necessary to determine whether the visual positioning data is abnormal. Only when the visual positioning data is normal can the next judgment on whether the motion threshold is reached be carried out.

[0111] Since too few or too many calibration points will cause the picture presented by the target AV&VR headset device to result in a poor user experience, in an embodiment of the present invention, 15 ultra-wideband calibration points are collected from the preprocessed ultra-wideband positioning data, and the ultra-wideband calibration point set is set as P UWB ={p i}, and when the visual positioning data reaches the motion threshold, 15 visual calibration points are collected from the visual positioning data, and the visual calibration point set is set as P Vision ={v i}, represents the real number field.

[0112] In an embodiment of the present invention, before converting the visual positioning data to the ultra-wideband coordinate system, it further includes:

[0113] Calculating the centroid of the ultra-wideband calibration points and the centroid of the visual calibration points respectively, and the expressions are:

[0114]

[0115] Among them, p cRepresents the centroid of the UWB calibration point, v c Represents the centroid of the visual calibration point.

[0116] Most preferably, the visual positioning data is converted to the UWB coordinate system, including:

[0117] Both the UWB calibration point and the visual calibration point are converted to the coordinate system centered on the centroid, obtaining the UWB calibration point in the coordinate system centered on the centroid and the visual calibration point in the coordinate system centered on the centroid. The expression is:

[0118] p′ i = p i - p c

[0119] v′ i = v i - v c

[0120] Among them, p' i Represents the UWB calibration point in the coordinate system centered on the centroid, and v' i Represents the visual calibration point in the coordinate system centered on the centroid;

[0121] Use the UWB calibration point in the coordinate system centered on the centroid and the visual calibration point in the coordinate system centered on the centroid to construct a cross-covariance matrix, and after performing singular value decomposition on the cross-covariance matrix, solve for the optimal rotation matrix to align the visual calibration point with the UWB calibration point as much as possible. The expression is:

[0122]

[0123] H = U∑V T

[0124] R = VU T

[0125] Among them, H represents the cross-covariance matrix, V' represents the centered point cloud matrix of the UWB calibration point, P' represents the centered point cloud matrix of the LOS calibration point, R represents the optimal rotation matrix, U represents an orthogonal matrix, and its column vectors (left singular vectors) form an orthogonal basis of the column space of H. These vectors reflect the main direction information in the UWB positioning data (target coordinate system); V T Is the transpose of V. The column vectors (right singular vectors) of V form an orthogonal basis of the row space of H. These vectors reflect the main direction information in the visual data (coordinate system to be converted); ∑ is a diagonal matrix of singular values, containing the singular values of H, which measures the variance of the point distribution in each direction, and U T Represents the transpose of U, which changes the column vectors of U into row vectors and is used to extract the basis in the UWB positioning data;

[0126] The expression for calculating the translation vector using the centroid of the ultra-wideband calibration points, the centroid of the visual calibration points, and the optimal rotation matrix is as follows:

[0127] T = p c - Rv c

[0128] where T represents the translation vector, and Rv c represents the centroid of the visual calibration points mapped to the ultra-wideband coordinate system;

[0129] Based on the translation vector and the optimal rotation matrix, the visual positioning data is converted to the ultra-wideband coordinate system.

[0130] Specifically, the conversion expression for converting the visual positioning data to the ultra-wideband coordinate system based on the translation vector and the optimal rotation matrix is:

[0131]

[0132] where represents the visual data in the ultra-wideband coordinate system, X vision represents the visual positioning data, T represents the translation vector, and R represents the optimal rotation matrix.

[0133] Specifically, the calculation expression for fusing the visual positioning data in the ultra-wideband coordinate system with the preprocessed ultra-wideband positioning data is:

[0134]

[0135] where x fused and y fused respectively represent the x-coordinate and y-coordinate of the positioning data after weighted average fusion, w UWB represents the weight of the preprocessed ultra-wideband positioning data, represents the position error variance of the ultra-wideband positioning data, w Vision represents the weight of the visual positioning data, represents the position error variance of the visual positioning data, x ukf and y ukf respectively represent the x-coordinate and y-coordinate of the preprocessed ultra-wideband positioning data, respectively represent the x-coordinate and y-coordinate of the visual positioning data in the ultra-wideband coordinate system.

[0136] Specifically, before obtaining the fused positioning data, it also includes:

[0137] Filter the positioning data after weighted average fusion through the UKF filtering algorithm, and use the filtered positioning data as the fused positioning data.

[0138] As Figure 3 shown, the method provided by the embodiment of the present invention can form a synchronous processing mechanism to fuse the ultra-wideband positioning data and visual positioning data collected by multiple IDs simultaneously. The synchronous processing mechanism includes a data fusion manager, multiple data processors, and multiple asynchronous data senders; the data fusion manager reads the data stored in the ultra-wideband queue and the visual queue, coordinates each data processor to process the ultra-wideband positioning data and visual positioning data of different IDs, provides a UKF filtering algorithm for each data processor for filtering, processes the data through each data processor and transmits the fused data to the corresponding asynchronous data sender. Each asynchronous data sender uses a non-blocking method to create an asynchronous sending task for each ID to send the fused positioning data to the corresponding target AV&VR headset device, ensuring the real-time performance and high concurrency performance of data sending.

[0139] The embodiment of the present invention reads the ultra-wideband positioning data and visual positioning data in an asynchronous manner; preprocesses the ultra-wideband positioning data to obtain the preprocessed ultra-wideband positioning data; detects the visual positioning data. When it is detected that the visual positioning data reaches the motion threshold, several ultra-wideband calibration points and visual calibration points are respectively collected from the preprocessed ultra-wideband positioning data and visual positioning data; when the number of ultra-wideband calibration points and visual calibration points reaches the preset number and the visual positioning data is completely converted to the ultra-wideband coordinate system, the visual data in the ultra-wideband coordinate system is fused with the preprocessed ultra-wideband positioning data to obtain the fused positioning data, and the fused positioning data is applied to the target AV&VR headset device; compared with the prior art, the present invention preprocesses the ultra-wideband positioning data and collects calibration points, collects calibration points for the visual positioning data, performs coordinate conversion on the visual positioning data based on the centroid of the calibration points and determines whether the conversion is completed. If the conversion is completed, the visual data in the ultra-wideband coordinate system is fused with the preprocessed ultra-wideband positioning data, otherwise, several ultra-wideband calibration points and visual calibration points are collected again, which can solve the problem that in the traditional head-mounted display device using the visual positioning mode, it is impossible or relatively cumbersome to perform multi-person relative position positioning in the multi-person large-space AR&VR content scene, and improves the interaction experience between people / virtual scenes in the multi-person large-space AR&VR content scene.

[0140] Corresponding to the multi-source positioning data fusion method described in the above embodiment, as Figure 4 shown, the embodiment of the present invention further provides a multi-source positioning data fusion device 100, and the multi-source positioning data fusion device 100 includes:

[0141] A reading module 101, configured to read the ultra-wideband positioning data and visual positioning data in an asynchronous manner;

[0142] The first processing module 102 is configured to preprocess the ultra-wideband positioning data to obtain the preprocessed ultra-wideband positioning data;

[0143] The second processing module 103 is configured to detect the visual positioning data. When it detects that the visual positioning data reaches the motion threshold, several ultra-wideband calibration points and visual calibration points are respectively collected from the preprocessed ultra-wideband positioning data and the visual positioning data;

[0144] The conversion module 104 is configured to, when the number of ultra-wideband calibration points and visual calibration points reaches the preset quantity, convert the visual positioning data to the ultra-wideband coordinate system and determine whether the conversion is completed;

[0145] If the conversion is completed, the visual data in the ultra-wideband coordinate system is fused with the preprocessed ultra-wideband positioning data to obtain the fused positioning data, and the fused positioning data is applied to the target AV&VR headset device;

[0146] If the conversion is not completed, several ultra-wideband calibration points and visual calibration points are collected again, and step 4 is executed again.

[0147] It should be noted that for the information interaction, execution process, etc. between the above-mentioned devices / units, since they are based on the same concept as the method embodiment of the present application, for their specific functions and the technical effects brought, reference can be specifically made to the method embodiment part, and details are not described herein again.

[0148] Those skilled in the art can clearly understand that for the convenience and conciseness of description, only the above-mentioned division of each functional unit and module is used as an example for illustration. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiment, and details are not described herein again.

[0149] An embodiment of the present invention further provides a terminal device, as Figure 5 shown. The terminal device D10 in this embodiment includes: at least one processor D100( Figure 5Only one processor is shown), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100. When the processor D100 executes the computer program D102, the above multi-source positioning data fusion method is implemented.

[0150] The terminal device D10 may be a computing device such as a desktop computer, a notebook, a palm computer, a server, a server cluster, and a cloud server. The terminal device may include, but is not limited to, a processor D100 and a memory D101. Those skilled in the art can understand that Figure 5 These are merely examples of the terminal device D10 and do not constitute a limitation on the terminal device D10. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0151] The so-called processor D100 may be a central processing unit (CPU, Central Processing Unit), and the processor D100 may also be other general-purpose processors, digital signal processors (DSP, Digital Signal Processor), application specific integrated circuits (ASIC, Application Specific Integrated Circuit), field-programmable gate arrays (FPGA, Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0152] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as the hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk equipped on the terminal device D10, a smart media card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. Further, the memory D101 may also include both the internal storage unit and the external storage device of the terminal device D10. The memory D101 is used to store an operating system, application programs, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or will be output.

[0153] It should be noted that, for the information interaction, execution process, etc. between the above-mentioned devices / units, since they are based on the same concept as the method embodiments of the present application, for their specific functions and the technical effects brought about, reference can be specifically made to the method embodiment part, and details will not be repeated here.

[0154] Those skilled in the art can clearly understand that, for the sake of convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiment, and details will not be repeated here.

[0155] The above is the preferred embodiment of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A multi-source positioning data fusion method, characterized in that: include: Step 1: read ultra-wideband positioning data and visual positioning data in an asynchronous manner; Step 2, preprocessing the ultra-wideband positioning data to obtain preprocessed ultra-wideband positioning data; Step 3, detecting the visual positioning data, and when it is detected that the visual positioning data reaches a motion threshold, collecting a number of ultra-wideband calibration points and visual calibration points in the preprocessed ultra-wideband positioning data and the visual positioning data respectively; Step 4, when the ultra-wideband calibration points and the visual calibration points reach a preset number, converting the visual positioning data to an ultra-wideband coordinate system and determining whether the conversion is completed; If the conversion is completed, the visual data in the ultra-wideband coordinate system is fused with the pre-processed ultra-wideband positioning data to obtain fused positioning data, and the fused positioning data is applied to the target AV&VR head display device; If the conversion is not completed, a number of ultra-wideband calibration points and visual calibration points are collected again, and the process returns to step 4.

2. The multi-source positioning data fusion method according to claim 1, characterized in that: The step 1 comprises: Continuously read multiple positioning frames sent by the ultra-wideband device through asynchronous serial communication; Parsing each positioning frame to obtain key attributes of each positioning frame as ultra-wideband positioning data, wherein the ultra-wideband positioning data is stored in an ultra-wideband queue; By parsing the positioning message sent by the asynchronous UDP remote monitoring system, three-dimensional coordinates and posture information are obtained as visual positioning data, and the visual positioning data is stored in the visual queue.

3. The multi-source positioning data fusion method according to claim 2, characterized in that: Preprocessing the ultra-wideband positioning data to obtain preprocessed ultra-wideband positioning data includes: Inputting the ultra-wideband positioning data into a UKF filtering algorithm; Define the state of the UKF filtering algorithm and construct the prediction equation, observation equation and covariance matrix; Generating a plurality of sigma points in the ultra-wideband positioning data, and substituting each sigma point into the prediction equation for solving, to obtain a priori prediction results; Calculate the prior mean and the prior covariance based on the prior prediction results; Substitute each prior sigma point into the observation equation for calculation to obtain the scale factor of the measurement space; Calculating a measurement prediction mean and a measurement prediction variance using a scale factor of the measurement space; Calculating a cross covariance based on the prior prediction result, the prior mean, a scaling factor of the measurement space, and the measurement prediction mean; Calculating a Kalman gain using the cross covariance and the measurement covariance; The posterior state mean is calculated based on the Kalman gain, the prior mean, and the measurement prediction mean, and the numbers of the first two dimensions in the posterior state mean are used as preprocessed ultra-wideband positioning data.

4. The multi-source positioning data fusion method according to claim 3, characterized in that: The expression for calculating the posterior state mean based on the Kalman gain, the prior mean, and the measurement prediction mean is: in, represents the posterior state mean, represents the prior mean, K k+1 represents the Kalman gain, z k+1 represents the actual measured value, represents the measured predicted mean.

5. The multi-source positioning data fusion method according to claim 4, characterized in that: Converting the visual positioning data to an ultra-wideband coordinate system includes: Converting the ultra-wideband calibration points and the visual calibration points to a coordinate system centered on the center of mass to obtain ultra-wideband calibration points in the coordinate system centered on the center of mass and visual calibration points in the coordinate system centered on the center of mass; Constructing a cross-covariance matrix using ultra-wideband calibration points in a coordinate system centered on the centroid and visual calibration points in a coordinate system centered on the centroid, and solving an optimal rotation matrix after performing singular value decomposition on the cross-covariance matrix; The translation vector is calculated using the centroid of the ultra-wideband calibration point, the centroid of the visual calibration point and the optimal rotation matrix; The visual positioning data is converted into an ultra-wideband coordinate system based on the translation vector and the optimal rotation matrix.

6. The multi-source positioning data fusion method according to claim 5, characterized in that: The conversion expression for converting the visual positioning data to the ultra-wideband coordinate system based on the translation vector and the optimal rotation matrix is: in, Represents the visual data in the ultra-wideband coordinate system, X vision Represents visual positioning data, T represents the translation vector, and R represents the optimal rotation matrix.

7. The multi-source positioning data fusion method according to claim 6, characterized in that: The calculation expression for fusing the visual data in the ultra-wideband coordinate system with the preprocessed ultra-wideband positioning data is: Among them, x fused ,y fused They represent the x-coordinate and y-coordinate of the positioning data after weighted average fusion, w UWB represents the weight of the preprocessed ultra-wideband positioning data, w Vision represents the weight of visual positioning data, x ukf ,y ukf They represent the x-coordinate and y-coordinate of the preprocessed ultra-wideband positioning data, They respectively represent the x-coordinate and y-coordinate of the visual positioning data in the ultra-wideband coordinate system.

8. The multi-source positioning data fusion method according to claim 7, characterized in that: Before obtaining the fused positioning data, it also includes: The weighted average fused positioning data is filtered by the UKF filtering algorithm, and the filtered positioning data is used as the fused positioning data.

9. A multi-source positioning data fusion device, characterized in that: include: A reading module, used for reading ultra-wideband positioning data and visual positioning data in an asynchronous manner; A first processing module is used to preprocess the ultra-wideband positioning data to obtain preprocessed ultra-wideband positioning data; A second processing module is used to detect the visual positioning data, and when it is detected that the visual positioning data reaches a motion threshold, collect a number of ultra-wideband calibration points and visual calibration points in the pre-processed ultra-wideband positioning data and the visual positioning data respectively; A conversion module, used for converting the visual positioning data into an ultra-wideband coordinate system and determining whether the conversion is completed when the ultra-wideband calibration points and the visual calibration points reach a preset number; If the conversion is completed, the visual data in the ultra-wideband coordinate system is fused with the pre-processed ultra-wideband positioning data to obtain fused positioning data, and the fused positioning data is applied to the target AV&VR head display device; If the conversion is not completed, a number of ultra-wideband calibration points and visual calibration points are collected again, and the process returns to step 4.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multi-source positioning data fusion method according to any one of claims 1 to 7 is implemented.

Images