Positioning method, positioning device and computer readable storage medium

Abstract

The present application discloses a positioning method, a positioning device and a computer-readable storage medium, wherein the positioning method includes: comparing the uncertainty of the decentralized transformation relationship between the cloud and the device end with a first preset threshold; wherein the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device end; in response to the uncertainty of the decentralized transformation relationship being greater than the first preset threshold, updating the decentralized transformation relationship according to the current cloud positioning information and the current device end tracking information matching the current cloud positioning information; and using the updated decentralized transformation relationship to fuse and locate the subsequent cloud positioning information and the device end tracking information. The above scheme can make the end-cloud fusion results obtained have better consistency and efficiency.

Description

Technical Field

[0001] The present application relates to the field of positioning technology, and in particular to a positioning method, a positioning device and a computer-readable storage medium. Background Art

[0002] Positioning and navigation based on pre-built maps are important underlying technologies in the fields of computer vision, robotics, unmanned vehicles, augmented reality, etc. The end-cloud fusion navigation solution with low communication cost, low power load, high stability and high coupling has important practical value in the fields of unmanned vehicles, robotics and augmented reality.

[0003] The difficulties of integrating device-side tracking with cloud-side positioning technology are: 1) Cloud-side positioning is affected by factors such as dynamic scenes and environmental changes, the positioning success rate is not high, and the positioning posture has a high uncertainty; 2) Due to the limitations of communication delay and server computing power, there will be a large delay when cloud-side positioning returns to the device side; 3) In order to reduce the inconsistency between cloud-side maps and device-side tracking and mapping information, the cloud and device sides need to exchange high-frequency and rich data, which also increases the communication load and the computing power load on both the end and the cloud. Summary of the invention

[0004] The present application provides at least one positioning method, a positioning device and a computer-readable storage medium.

[0005] The first aspect of the present application provides a positioning method, the method comprising: comparing the uncertainty of a decentralized transformation relationship between a cloud and a device with a first preset threshold; wherein the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device; in response to the uncertainty of the decentralized transformation relationship being greater than the first preset threshold, updating the decentralized transformation relationship according to current cloud positioning information and current device tracking information matching the current cloud positioning information; and using the updated decentralized transformation relationship to fuse and locate subsequent cloud positioning information and device tracking information.

[0006] Therefore, the cloud positioning information and the device-side tracking information are fused and positioned through the decentralized transformation relationship between the cloud and the device, which has low requirements on the computing power and data communication of the device, thereby realizing efficient end-cloud fusion, and the obtained end-cloud fusion results have good consistency; in addition, when the uncertainty of the decentralized transformation relationship is greater than the first preset threshold, the decentralized transformation relationship is updated, and the updated decentralized transformation relationship is used to fuse and position the subsequent cloud positioning information and the subsequent device-side tracking information, so that the end-cloud fusion results are highly accurate.

[0007] Among them, before the step of comparing the uncertainty of the decentralized transformation relationship between the cloud and the device side with a first preset threshold, the method includes: obtaining initial cloud positioning information; initializing the decentralized transformation relationship according to the initial cloud positioning information and initial device-side tracking information matching the initial cloud positioning information.

[0008] Therefore, by initializing the decentralized transformation relationship according to the initial cloud positioning information and the initial device-side tracking information that matches the initial cloud positioning information, an initialized decentralized transformation relationship can be obtained, so that the cloud positioning information and the device-side tracking information can be integrated and positioned through the decentralized transformation relationship between the cloud and the device.

[0009] The method further includes: obtaining the uncertainty of the initialized decentralized transformation relationship based on the cloud mapping error and the observation error when the initial cloud positioning information is matched with the initial device-side tracking information.

[0010] Therefore, by comprehensively considering the cloud mapping error and the observation error when matching the initial cloud positioning information with the initial device-side tracking information, the uncertainty of the initialized decentralized transformation relationship can be obtained. This allows the decentralized transformation relationship to be updated when the uncertainty of the decentralized transformation relationship is greater than a first preset threshold, thereby making the end-cloud fusion result more accurate.

[0011] Wherein, the method includes: updating the uncertainty of the decentralized transformation relationship according to the accumulated error of the device end during operation.

[0012] Therefore, by updating the uncertainty of the decentralized transformation relationship according to the accumulated errors on the device side during operation, the uncertainty of the decentralized transformation relationship can be used to reflect the degree of inaccuracy caused by the accumulated errors to the decentralized transformation relationship. Therefore, when the uncertainty of the decentralized transformation relationship is greater than the first preset threshold, the decentralized transformation relationship can be updated, thereby making the end-cloud fusion result more accurate.

[0013] Among them, updating the uncertainty of the decentralized transformation relationship according to the accumulated error of the device end during the operation includes: updating the uncertainty of the decentralized transformation relationship with the covariance matrix of the posture information detected by the device end during the operation.

[0014] Therefore, since the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device, the independently developed device-side tracking and positioning algorithm, that is, the device-side white box tracking and positioning algorithm, can not only obtain the output results of each state quantity, but also obtain the corresponding confidence and covariance matrix and other information. Therefore, the uncertainty of the decentralized transformation relationship can be directly updated through the covariance matrix of the posture information, so that when the uncertainty of the decentralized transformation relationship is greater than the first preset threshold, the decentralized transformation relationship is updated, thereby making the end-cloud fusion result more accurate.

[0015] Wherein, updating the uncertainty of the decentralized transformation relationship according to the accumulated error of the device end during operation includes: updating the uncertainty of the decentralized transformation relationship at a preset error accumulation rate.

[0016] Therefore, since the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device, for the general device-side tracking and positioning algorithm, that is, the device-side black box tracking and positioning algorithm, the uncertainty of the decentralized transformation relationship can be updated through a preset error accumulation rate, so that when the uncertainty of the decentralized transformation relationship is greater than the first preset threshold, the decentralized transformation relationship is updated, thereby making the end-cloud fusion result highly accurate.

[0017] The preset error accumulation rate is positively correlated with the tracking duration or tracking distance of the device.

[0018] Therefore, by setting a preset error accumulation rate that is positively correlated with the tracking time or tracking distance on the device side, and updating the uncertainty of the decentralized transformation relationship at the preset error accumulation rate, the accumulated error generated by the black box system on the device side during the movement process can be incorporated into the positioning method, thereby making the end-cloud fusion result more accurate.

[0019] Among them, the map used in the cloud includes several sub-maps, and there is a preset map association relationship between any two sub-maps. The decentralized transformation relationship includes a first transformation relationship between the global coordinate system of the sub-map observed during initialization and the local coordinate system of the device end, and a second transformation relationship between the global coordinate system of the unobserved sub-map calculated by the first transformation relationship and the preset map association relationship and the local coordinate system of the device end.

[0020] Therefore, when the map used in the cloud includes several sub-maps, the second transformation relationship between the global coordinate system of the observed sub-map and the local coordinate system of the device side can be obtained through the first transformation relationship between the global coordinate system of the observed sub-map and the local coordinate system of the device side, and the preset map association relationship between any two sub-maps. Therefore, the decentralized transformation relationship may include the first association relationship and the second association relationship, so that even if a sub-map is not observed within a certain period of time, the real-time fusion positioning result of the device-side tracking and positioning algorithm in the global coordinate system of the unobserved sub-map can still be obtained relatively stably.

[0021] Among them, before the step of using the updated decentralized transformation relationship to fuse and locate the subsequent cloud positioning information and the device-side tracking information, the method also includes: projecting the subsequent cloud positioning information of the cloud onto the local coordinate system of the device side through the decentralized transformation relationship before updating and the decentralized transformation relationship after updating, respectively, to obtain a first reprojection value and a second reprojection value; comparing the error between the first reprojection value and the second reprojection value with a second preset threshold; in response to the error between the first reprojection value and the second reprojection value being greater than the second preset threshold, executing the step of using the updated decentralized transformation relationship to fuse and locate the subsequent cloud positioning information and the device-side tracking information; in response to the error between the first reprojection value and the second reprojection value being not greater than the second preset threshold, using the decentralized transformation relationship before updating to fuse and locate the subsequent cloud positioning information and the device-side tracking information.

[0022] Therefore, the subsequent cloud positioning information in the cloud is projected onto the local coordinate system of the device side through the decentralized transformation relationship before updating and the decentralized transformation relationship after updating, so as to obtain the first reprojection value and the second reprojection value, and it is determined whether the error between the first reprojection value and the second reprojection value is greater than the second preset threshold value. If so, the updated decentralized transformation relationship is used to fuse and locate the subsequent cloud positioning information and the subsequent device-side tracking information. If not, the decentralized transformation relationship before updating is used to fuse and locate the subsequent cloud positioning information and the subsequent device-side tracking information, so that when the error between the first reprojection value and the second reprojection value is small, it is chosen not to adjust the decentralized transformation relationship, which will not cause frequent jumps in the fusion positioning result, so that users have a better user experience.

[0023] Wherein, before executing the step of adopting the updated decentralized transformation relationship to fuse and locate the subsequent cloud positioning information and the device-side tracking information, the method further includes: comparing the error between the first reprojection value and the second reprojection value corresponding to the subsequent cloud positioning information of the cloud for several consecutive frames with a second preset threshold; in response to the error between the first reprojection value and the second reprojection value corresponding to the subsequent cloud positioning information of the cloud for several consecutive frames being greater than the second preset threshold, reinitializing the decentralized transformation relationship according to the latest subsequent cloud positioning information of the cloud and the device-side tracking information matching the latest subsequent cloud positioning information of the cloud, and adopting the reinitialized decentralized transformation relationship to fuse and locate the subsequent cloud positioning information and the device-side tracking information; in response to the error between the first reprojection value and the second reprojection value corresponding to the subsequent cloud positioning information of the cloud for several consecutive frames being greater than the second preset threshold, executing the step of adopting the updated decentralized transformation relationship to fuse and locate the subsequent cloud positioning information and the device-side tracking information.

[0024] Therefore, by judging whether the errors between the first reprojection value and the second reprojection value corresponding to the subsequent cloud positioning information of several consecutive frames of the cloud are both greater than the second preset threshold value, if so, the decentralized transformation relationship is reinitialized according to the latest subsequent cloud positioning information of the cloud and the device-side tracking information matching the latest subsequent cloud positioning information of the cloud, and the reinitialized decentralized transformation relationship is used to fuse and locate the subsequent cloud positioning information and the subsequent device-side tracking information; if not, the updated decentralized transformation relationship is used to fuse and locate the subsequent cloud positioning information and the subsequent device-side tracking information; the decentralized transformation relationship can be reestablished when the system error is too large, thereby making the end-cloud fusion result highly accurate.

[0025] The device end has at least two different operating stages, wherein the second preset threshold values ​​in different operating stages are different.

[0026] Therefore, since users have relatively different requirements for the stability and accuracy of the fusion positioning results in different operating stages, by setting different second preset thresholds for different operating stages, the needs of users in different operating stages can be met, thereby improving the user experience.

[0027] In order to solve the above problems, the second aspect of the present application provides a positioning device, including: a judgment module, used to judge whether the uncertainty of the decentralized transformation relationship between the cloud and the device end is greater than a first preset threshold, wherein the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device end; an update module, used to update the decentralized transformation relationship according to the current cloud positioning information and the current device-side tracking information matching the current cloud positioning information when judging whether the uncertainty of the decentralized transformation relationship between the cloud and the device end is greater than the first preset threshold; a positioning module, used to use the updated decentralized transformation relationship to fuse and locate the subsequent cloud positioning information and the device-side tracking information.

[0028] In order to solve the above problem, the third aspect of the present application provides a positioning device, including a memory and a processor coupled to each other, and the processor is used to execute program instructions stored in the memory to implement the positioning method in the first aspect.

[0029] In order to solve the above problem, the fourth aspect of the present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the positioning method in the above first aspect is implemented.

[0030] The above scheme determines whether the uncertainty of the decentralized transformation relationship between the cloud and the device is greater than a first preset threshold, wherein the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device, and when it is determined that the uncertainty of the decentralized transformation relationship between the cloud and the device is greater than the first preset threshold, the decentralized transformation relationship is updated according to the current cloud positioning information and the current device tracking information that matches the current cloud positioning information, and then the updated decentralized transformation relationship is used to fuse and locate the subsequent cloud positioning information and the subsequent device tracking information. Therefore, the cloud positioning information and the device tracking information are fused and located by the decentralized transformation relationship between the cloud and the device, which has low requirements on the computing power and data communication of the device, thereby realizing efficient end-cloud fusion, and the end-cloud fusion results obtained have good consistency; in addition, when the uncertainty of the decentralized transformation relationship is greater than the first preset threshold, the decentralized transformation relationship is updated, and the updated decentralized transformation relationship is used to fuse and locate the subsequent cloud positioning information and the subsequent device tracking information, so that the end-cloud fusion result is highly accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flowchart of an embodiment of the positioning method of the present application;

[0032] Figure 2 is a flow chart of another embodiment of the positioning method of the present application;

[0033] Figure 3 It is a flowchart of another embodiment of the positioning method of the present application;

[0034] Figure 4 It is a flowchart of another embodiment of the positioning method of the present application;

[0035] Figure 5 It is a schematic diagram of the framework of an embodiment of the positioning device of the present application;

[0036] Figure 6 is a schematic diagram of a framework of another embodiment of the positioning device of the present application;

[0037] Figure 7 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0038] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0039] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0040] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0041] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the positioning method of the present application. Specifically, it may include the following steps:

[0042] Step S11: Compare the uncertainty of the decentralized transformation relationship between the cloud and the device with the first preset threshold; wherein the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device. It is understandable that in response to the uncertainty of the decentralized transformation relationship being greater than the first preset threshold, step S12 is executed; in response to the uncertainty of the decentralized transformation relationship being not greater than the first preset threshold, it is indicated that the decentralized transformation relationship is currently more accurate. In order to avoid frequent jumps in the fusion positioning results, there is no need to update the decentralized transformation relationship temporarily, and the process ends.

[0043] In the process of integrated positioning between the tracking system on the device side and the positioning system on the cloud side, the device side and the cloud side do not share the same coordinate system. Therefore, it is necessary to establish a decentralized transformation relationship to realize the coordinate transformation between the global coordinate system of the cloud side and the local coordinate system of the device side. The decentralized transformation relationship can transform a frame of cloud-side positioning pose information with a corresponding frame of device-side tracking pose information. Through this decentralized transformation relationship, the relative relationship between the cloud-side positioning information and the matching device-side tracking information can be maintained.

[0044] In one embodiment, the expression form of the decentralized transformation relationship can be selected according to the information of the cloud map and the information tracked on the device. When the scale of the cloud map and the device tracking is consistent, the decentralized transformation relationship can be a rigid transformation with six degrees of freedom; when the scale of the cloud map and the device tracking is inconsistent, the decentralized transformation relationship can be a similar transformation with seven degrees of freedom; when the scale of the cloud map and the device tracking is consistent, and the gravity directions of the cloud map and the device are both observable, the decentralized transformation relationship can be a position plus yaw angle transformation with four degrees of freedom; when the scale of the cloud map and the device tracking is consistent, and the gravity directions and yaw angles of the cloud map and the device are both observable, the decentralized transformation relationship can be a position transformation with three degrees of freedom.

[0045] It can be understood that in the process of tracking and positioning, the accuracy of the decentralized transformation relationship is indicated by adding an uncertainty to the decentralized transformation relationship; when the uncertainty of the decentralized transformation relationship between the cloud and the device is greater than the first preset threshold, it means that the accuracy of the decentralized transformation relationship is poor, and the decentralized transformation relationship needs to be updated at this time; when the uncertainty of the decentralized transformation relationship between the cloud and the device is not greater than the first preset threshold, it means that the decentralized transformation relationship is currently more accurate, so the decentralized transformation relationship does not need to be updated.

[0046] Step S12: updating the decentralized transformation relationship according to the current cloud positioning information and the current device-side tracking information matching the current cloud positioning information.

[0047] It is understandable that after obtaining the current cloud positioning information, it can be compared with all the tracking information on the current device side, and then the current device side tracking information matching the current cloud positioning information can be obtained. At this time, according to the current cloud positioning information and the current device side tracking information matching the current cloud positioning information, the decentralized transformation relationship between the cloud and the device side can be determined, that is, the decentralized transformation relationship can be optimized and updated. In this embodiment, the decentralized transformation relationship can be optimized and updated by using methods such as extended Kalman filter, unscented Kalman filter, root mean square filter, nonlinear optimization, etc.

[0048] In one embodiment, since the positioning information of a single frame has randomness and a large amount of noise, the decentralized transformation relationship can be updated based on multiple frames of current cloud positioning information and multiple frames of current device-side tracking information that match the multiple frames of current cloud positioning information, so that the end-cloud fusion result is highly accurate.

[0049] Step S13: Use the updated decentralized transformation relationship to integrate the subsequent cloud positioning information and device-side tracking information.

[0050] After the decentralized transformation relationship is updated, in the subsequent fusion positioning process, the device side initiates a receive request and receives the subsequent cloud positioning information sent by the cloud. It can be compared with all the tracking information on the device side to obtain the subsequent device-side tracking information that matches the subsequent cloud positioning information. Then, the updated decentralized transformation relationship is used to fuse the subsequent cloud positioning information and the subsequent device-side tracking information.

[0051] The execution subject of the positioning method of the present application can be a positioning device, for example, the positioning method can be executed by a terminal device or a cloud device or a server or other processing device, wherein the terminal device can be a mobile device such as a robot, an unmanned vehicle, or a drone, or a user equipment (User Equipment, UE), a user terminal, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the positioning method can be implemented by a processor calling a computer-readable instruction stored in a memory. The device-side tracking and positioning algorithm of the present application can use a device-side preset tracking and positioning algorithm such as ARKit, ARCore, or a visual inertial tracking and positioning algorithm based on the image and inertial navigation information of the device side for tracking and positioning, or a pure visual tracking and positioning algorithm based on single / dual / multi-eye image information.

[0052] The above scheme integrates and locates the cloud positioning information and the device-side tracking information through the decentralized transformation relationship between the cloud and the device side, and has low requirements on the computing power and data communication of the device side, thereby realizing efficient end-cloud integration, and the end-cloud fusion results obtained have good consistency; in addition, when the uncertainty of the decentralized transformation relationship is greater than the first preset threshold, the decentralized transformation relationship is optimized and updated, and the updated decentralized transformation relationship is used to integrate and locate the subsequent cloud positioning information and the subsequent device-side tracking information, so that the end-cloud fusion results are highly accurate.

[0053] See also Figure 2 , Figure 2 This is a flow chart of another embodiment of the positioning method of the present application. Specifically, it may include the following steps:

[0054] Step S21: Obtain initial cloud positioning information.

[0055] Step S22: Initializing the decentralized transformation relationship according to the initial cloud positioning information and the initial device-side tracking information matching the initial cloud positioning information.

[0056] It is understandable that, since there are large delays in data communication and cloud positioning between the cloud and the device, the device tracking information can be pre-saved. After the initial cloud positioning information is obtained, it can be compared with all the tracking information on the device, and then the initial device tracking information that matches the initial cloud positioning information can be obtained. At this time, based on the initial cloud positioning information and the initial device tracking information that matches the initial cloud positioning information, the decentralized transformation relationship between the cloud and the device can be determined, that is, the decentralized transformation relationship can be initialized. Specifically, after the first frame image with successful cloud positioning is obtained, it is compared with all the key frames of the device tracking information. If the first frame image with successful cloud positioning matches a key frame of the device tracking information, the pose information of the first frame image with successful cloud positioning and the pose information corresponding to the key frame of the device tracking information can be obtained. Therefore, the initialized decentralized transformation relationship between the cloud and the device can be determined, so that the cloud positioning information and the device tracking information can be integrated and positioned through the decentralized transformation relationship between the cloud and the device.

[0057] Furthermore, the positioning method of the present application may also include: obtaining the uncertainty of the initialized decentralized transformation relationship based on the cloud mapping error and the observation error when the initial cloud positioning information is matched with the initial device-side tracking information. It can be understood that since the cloud positioning result has a large noise, the uncertainty of the initialized decentralized transformation relationship needs to be derived from the uncertainty of the cloud positioning. Specifically, the uncertainty of cloud positioning includes at least two aspects of uncertainty. The first aspect of uncertainty is that there is a mapping error when the cloud map is constructed. The second aspect of uncertainty is that during positioning, there is an observation error when matching the three-dimensional points in the initial cloud positioning information to the two-dimensional points of the initial device-side tracking information. Combining these two aspects of uncertainty, the initialized decentralized transformation relationship can be obtained, so that when the uncertainty of the decentralized transformation relationship is greater than the first preset threshold, the decentralized transformation relationship can be updated, thereby making the end-cloud fusion result highly accurate.

[0058] Step S23: Compare the uncertainty of the decentralized transformation relationship between the cloud and the device with a first preset threshold; wherein the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device. In response to the uncertainty of the decentralized transformation relationship being greater than the first preset threshold, step S24 is executed; in response to the uncertainty of the decentralized transformation relationship being not greater than the first preset threshold, the process ends.

[0059] Step S24: updating the decentralized transformation relationship according to the current cloud location information and the current device-side tracking information matching the current cloud location information.

[0060] Step S25: updating the uncertainty of the decentralized transformation relationship according to the accumulated error of the device during operation.

[0061] The device-side tracking and positioning system will produce error accumulation during the tracking process, and the error accumulation process needs to be included in the fusion positioning method. It is understandable that due to error accumulation, the confidence of the covariance matrix of the visual inertial system is getting lower and lower, that is, the decentralized transformation relationship is becoming less and less accurate due to the accumulated errors generated by the visual inertial system. Therefore, it is necessary to update the uncertainty of the decentralized transformation relationship according to the accumulated errors during the operation of the device side, so that the uncertainty of the decentralized transformation relationship can be used to reflect the inaccuracy caused by the accumulated errors to the decentralized transformation relationship. Therefore, when the uncertainty of the decentralized transformation relationship is greater than the first preset threshold, the decentralized transformation relationship can be updated, thereby making the end-cloud fusion result more accurate.

[0062] In one embodiment, the above step S25 specifically includes: updating the uncertainty of the decentralized transformation relationship with the covariance matrix of the posture information detected by the device during operation. It can be understood that since the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device, for the independently developed device-side tracking and positioning algorithm, that is, the device-side white box tracking and positioning algorithm, not only can the output results of each state quantity be obtained, but also the corresponding confidence and covariance matrix and other information can be obtained. Therefore, the uncertainty of the decentralized transformation relationship can be directly updated through the covariance matrix of the posture information of the device, so that when the uncertainty of the decentralized transformation relationship is greater than the first preset threshold, the decentralized transformation relationship is updated, thereby making the end-cloud fusion result highly accurate.

[0063] In one embodiment, the above step S25 specifically includes: updating the uncertainty of the decentralized transformation relationship at a preset error accumulation rate. It can be understood that since the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device, for the existing device-side tracking and positioning algorithm, that is, the device-side black box tracking and positioning algorithm, since only the output results of each state quantity can be obtained, but the corresponding confidence and covariance matrix and other information cannot be obtained, the uncertainty of the decentralized transformation relationship can be updated at a preset error accumulation rate, so that when the uncertainty of the decentralized transformation relationship is greater than the first preset threshold, the decentralized transformation relationship is updated, thereby making the end-cloud fusion result highly accurate.

[0064] Furthermore, the above-mentioned preset error accumulation rate is positively correlated with the tracking time or tracking distance of the device. It can be understood that the longer the tracking time of the device, or the longer the tracking distance of the device, the more the error accumulates. Therefore, by setting a preset error accumulation rate that is positively correlated with the tracking time or tracking distance of the device, and updating the uncertainty of the decentralized transformation relationship at the preset error accumulation rate, the accumulated error generated by the device-side black box system during the movement process can be incorporated into the positioning method, thereby making the end-cloud fusion result more accurate.

[0065] Step S26: using the updated decentralized transformation relationship to perform fusion positioning on subsequent cloud positioning information and device-side tracking information.

[0066] In this embodiment, steps S23, S24, and S26 are basically similar to steps S11-S13 of the above-mentioned embodiment of the present application, and will not be repeated here.

[0067] In one embodiment, the map used in the cloud includes several sub-maps, and there is a preset map association relationship between any two sub-maps. The decentralized transformation relationship includes a first transformation relationship between the global coordinate system of the sub-map observed at the time of initialization and the local coordinate system of the device end, and a second transformation relationship between the global coordinate system of the unobserved sub-map and the local coordinate system of the device end calculated by the first transformation relationship and the preset map association relationship. It can be understood that in the case where the map used in the cloud includes several sub-maps, the second transformation relationship between the global coordinate system of the unobserved sub-map and the local coordinate system of the device end can be obtained through the first transformation relationship between the global coordinate system of the observed sub-map and the local coordinate system of the device end, and the preset map association relationship between any two sub-maps. Therefore, the present application can simultaneously maintain the transformation relationship between the global coordinate system of multiple sub-maps and the local coordinate system of the device end, that is, the decentralized transformation relationship can include the first association relationship and the second association relationship. Since the device-side tracking and positioning algorithm has good consistency in the short term, even if a sub-map is not observed for a certain period of time, the device-side tracking information and decentralized transformation relationship can still be used to obtain the real-time fusion positioning result of the device-side tracking information in the global coordinate system of the unobserved sub-map in a relatively stable manner.

[0068] See also Figure 3 , Figure 3 This is a flow chart of another embodiment of the positioning method of the present application. Specifically, it may include the following steps:

[0069] Step S31: Compare the uncertainty of the decentralized transformation relationship between the cloud and the device with a first preset threshold; wherein the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device. In response to the uncertainty of the decentralized transformation relationship being greater than the first preset threshold, step S32 is executed; in response to the uncertainty of the decentralized transformation relationship being not greater than the first preset threshold, the process ends.

[0070] Step S32: updating the decentralized transformation relationship according to the current cloud location information and the current device-side tracking information matching the current cloud location information.

[0071] Step S33: projecting subsequent cloud positioning information on the cloud onto the local coordinate system of the device through the decentralized transformation relationship before updating and the decentralized transformation relationship after updating, respectively, to obtain a first reprojection value and a second reprojection value.

[0072] Step S34: Compare the error between the first reprojection value and the second reprojection value with a second preset threshold. In response to the error between the first reprojection value and the second reprojection value being not greater than the second preset threshold, execute step S35; in response to the error between the first reprojection value and the second reprojection value being greater than the second preset threshold, execute step S36.

[0073] Step S35: Use the decentralized transformation relationship before updating to integrate the subsequent cloud positioning information and device-side tracking information.

[0074] After the decentralized transformation relationship is updated and adjusted, if the updated decentralized transformation relationship is directly used to fuse and locate the subsequent cloud positioning information and the subsequent device-side tracking information, the frequent update of the decentralized transformation relationship may cause frequent jumps in the fusion positioning result, thereby affecting the user experience; therefore, in this embodiment, the subsequent cloud positioning information of the cloud is projected onto the local coordinate system of the device side through the decentralized transformation relationship before the update and the decentralized transformation relationship after the update, so as to obtain a first reprojection value and a second reprojection value, and determine whether the error between the first reprojection value and the second reprojection value is greater than the second preset threshold. If so, the updated decentralized transformation relationship is used to fuse and locate the subsequent cloud positioning information and the subsequent device-side tracking information. If not, the decentralized transformation relationship before the update is used to fuse and locate the subsequent cloud positioning information and the subsequent device-side tracking information, so that when the error between the first reprojection value and the second reprojection value is small, it is selected not to adjust the decentralized transformation relationship, which will not cause frequent jumps in the fusion positioning result, so that the user has a better user experience.

[0075] Step S36: Use the updated decentralized transformation relationship to integrate the subsequent cloud positioning information and device-side tracking information.

[0076] In this embodiment, steps S31, S32, and S36 are basically similar to steps S11-S13 of the above-mentioned embodiment of the present application, and will not be repeated here.

[0077] See also Figure 4 , Figure 4 This is a flow chart of another embodiment of the positioning method of the present application. Specifically, it may include the following steps:

[0078] Step S41: Compare the uncertainty of the decentralized transformation relationship between the cloud and the device with a first preset threshold; wherein the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device. In response to the uncertainty of the decentralized transformation relationship being greater than the first preset threshold, step S42 is executed; in response to the uncertainty of the decentralized transformation relationship being not greater than the first preset threshold, the process ends.

[0079] Step S42: updating the decentralized transformation relationship according to the current cloud location information and the current device-side tracking information matching the current cloud location information.

[0080] Step S43: projecting subsequent cloud positioning information on the cloud onto the local coordinate system of the device through the decentralized transformation relationship before updating and the decentralized transformation relationship after updating, to obtain a first reprojection value and a second reprojection value.

[0081] Step S44: Compare the error between the first reprojection value and the second reprojection value with a second preset threshold. In response to the error between the first reprojection value and the second reprojection value being not greater than the second preset threshold, execute step S45; in response to the error between the first reprojection value and the second reprojection value being greater than the second preset threshold, execute step S46.

[0082] Step S45: Use the decentralized transformation relationship before updating to integrate the subsequent cloud positioning information and device-side tracking information.

[0083] Step S46: Compare the error between the first reprojection value and the second reprojection value corresponding to the subsequent cloud positioning information of the consecutive frames of cloud with the second preset threshold. In response to the error between the first reprojection value and the second reprojection value corresponding to the subsequent cloud positioning information of the consecutive frames of cloud being greater than the second preset threshold, step S47 is executed; in response to the error between the first reprojection value and the second reprojection value corresponding to the subsequent cloud positioning information of the consecutive frames of cloud being greater than the second preset threshold, step S48 is executed.

[0084] Step S47: Reinitialize the decentralized transformation relationship based on the latest cloud-based subsequent cloud positioning information and the device-side tracking information that matches the latest cloud-based subsequent cloud positioning information, and use the reinitialized decentralized transformation relationship to fuse and locate the subsequent cloud-based positioning information and the device-side tracking information.

[0085] It can be understood that when the subsequent cloud positioning information of several consecutive frames of the cloud is projected onto the local coordinate system of the device side through the decentralized transformation relationship before the update and the decentralized transformation relationship after the update, and the errors between the obtained first reprojection value and the second reprojection value are both greater than the second preset threshold, it may be that the error of the visual inertial system is too large. At this time, it is necessary to make a major adjustment to the decentralized transformation relationship, that is, to reinitialize the decentralized transformation relationship; that is, use the pose information of the latest frame image of the cloud positioning and the pose information of the key frame of the device-side tracking information matching the latest frame image of the cloud positioning to calculate the new decentralized transformation relationship. If there is a frame in the historical cloud positioning information whose reprojection error under the new decentralized transformation relationship is less than the second preset threshold, the decentralized transformation relationship is reinitialized, and the reinitialized decentralized transformation relationship is used to fuse and locate the subsequent cloud positioning information and the subsequent device-side tracking information. Therefore, the decentralized transformation relationship can be re-established when the system error is too large, thereby making the end-cloud fusion result highly accurate.

[0086] Step S48: using the updated decentralized transformation relationship to integrate and locate the subsequent cloud-based positioning information and device-side tracking information.

[0087] In this embodiment, steps S41-S45 and S48 are basically similar to steps S31-S36 of the above-mentioned embodiment of the present application, and will not be repeated here.

[0088] In one implementation scenario, the device side has at least two different operating stages, wherein the second preset thresholds of different operating stages are different. Specifically, the operation process of the device-side tracking and positioning system can be divided into different stages, such as the initialization stage, the moving navigation stage, and the stationary observation stage. In each stage, the user's requirements for the stability and accuracy of the fusion positioning are quite different, so it is necessary to adopt different second preset thresholds in each stage to improve the user experience. It is understandable that since the user's requirements for the stability and accuracy of the fusion positioning results are quite different in different operating stages, by setting different second preset thresholds for different operating stages, the user's needs in different operating stages can be met, thereby improving the user experience.

[0089] See also Figure 5 , Figure 5It is a schematic diagram of the framework of an embodiment of the positioning device of the present application. The positioning device 50 includes: a judgment module 500, which is used to judge whether the uncertainty of the decentralized transformation relationship between the cloud and the device end is greater than a first preset threshold, wherein the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device end; an update module 502, which is used to update the decentralized transformation relationship according to the current cloud positioning information and the current device end tracking information matching the current cloud positioning information when judging whether the uncertainty of the decentralized transformation relationship between the cloud and the device end is greater than the first preset threshold; a positioning module 504, which is used to use the updated decentralized transformation relationship to fuse the subsequent cloud positioning information and the device end tracking information.

[0090] In the above scheme, the positioning module 504 integrates and locates the cloud positioning information and the device-side tracking information through the decentralized transformation relationship between the cloud and the device, which has low requirements on the computing power and data communication of the device, thereby realizing efficient end-cloud fusion, and the obtained end-cloud fusion results have good consistency; in addition, when the uncertainty of the decentralized transformation relationship is greater than the first preset threshold, the judgment module 500 updates the decentralized transformation relationship through the update module 502, and uses the updated decentralized transformation relationship to integrate and locate the subsequent cloud positioning information and the subsequent device-side tracking information, so that the end-cloud fusion results are highly accurate.

[0091] In some embodiments, the positioning device 50 also includes an initialization module (not shown), which is specifically used to obtain initial cloud positioning information and initialize the decentralized transformation relationship according to the initial cloud positioning information and initial device-side tracking information matching the initial cloud positioning information.

[0092] In some embodiments, the updating module 502 is further used to update the uncertainty of the decentralized transformation relationship according to the accumulated error of the device end during operation.

[0093] In some embodiments, the updating module 502 executes the step of updating the uncertainty of the decentralized transformation relationship according to the accumulated error of the device end during operation, including: updating the uncertainty of the decentralized transformation relationship with the covariance matrix of the posture information detected by the device end during operation; or, updating the uncertainty of the decentralized transformation relationship at a preset error accumulation rate.

[0094] In some embodiments, the judgment module 500 is also used to project the subsequent cloud positioning information of the cloud onto the local coordinate system of the device end through the decentralized transformation relationship before the update and the decentralized transformation relationship after the update, respectively, to obtain a first reprojection value and a second reprojection value, and to determine whether the error between the first reprojection value and the second reprojection value is greater than a second preset threshold. If the judgment module 500 determines that the error between the first reprojection value and the second reprojection value is greater than the second preset threshold, the positioning module 504 executes the step of using the updated decentralized transformation relationship to fuse and locate the subsequent cloud positioning information and the device end tracking information; if the judgment module 500 determines that the error between the first reprojection value and the second reprojection value is not greater than the second preset threshold, the positioning module 504 is used to use the decentralized transformation relationship before the update to fuse and locate the subsequent cloud positioning information and the device end tracking information.

[0095] In some embodiments, the judgment module 500 is further used to judge whether the errors between the first reprojection value and the second reprojection value corresponding to the subsequent cloud positioning information of the cloud for several consecutive frames are all greater than the second preset threshold. If the judgment module 500 judges that the errors between the first reprojection value and the second reprojection value corresponding to the subsequent cloud positioning information of the cloud for several consecutive frames are all greater than the second preset threshold, the initialization module is used to reinitialize the decentralized transformation relationship according to the latest subsequent cloud positioning information of the cloud and the device-side tracking information matching the latest subsequent cloud positioning information of the cloud, and the positioning module 504 uses the reinitialized decentralized transformation relationship to fuse the subsequent cloud positioning information and the device-side tracking information; if the judgment module 500 judges that the errors between the first reprojection value and the second reprojection value corresponding to the subsequent cloud positioning information of the cloud for several consecutive frames are not all greater than the second preset threshold, the positioning module 504 executes the step of using the updated decentralized transformation relationship to fuse the subsequent cloud positioning information and the device-side tracking information.

[0096] See also Figure 6 , Figure 6 60 includes a memory 61 and a processor 62 coupled to each other, and the processor 62 is used to execute program instructions stored in the memory 61 to implement the steps of any of the above positioning method embodiments. In a specific implementation scenario, the positioning device 60 may include but is not limited to: a microcomputer, a server.

[0097] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps in any of the above-mentioned end-cloud fusion positioning embodiments. The processor 62 may also be referred to as a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip having signal processing capabilities. The processor 62 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 62 may be implemented by an integrated circuit chip.

[0098] In the above scheme, the processor 62 integrates and locates the cloud positioning information and the device-side tracking information through the decentralized transformation relationship between the cloud and the device, which has low requirements on the computing power and data communication of the device, thereby realizing efficient end-cloud fusion, and the obtained end-cloud fusion results have good consistency; in addition, when the uncertainty of the decentralized transformation relationship is greater than the first preset threshold, the decentralized transformation relationship is updated, and the updated decentralized transformation relationship is used to integrate and locate the subsequent cloud positioning information and the subsequent device-side tracking information, so that the end-cloud fusion results are highly accurate.

[0099] See also Figure 7 , Figure 7 The computer-readable storage medium 70 stores program instructions 700 that can be executed by a processor, and the program instructions 700 are used to implement the steps in any of the above-mentioned terminal-cloud fusion positioning embodiments.

[0100] The present disclosure relates to the field of augmented reality, which obtains image information of target objects in a real environment, and then uses various vision-related algorithms to detect or identify the relevant features, states and attributes of the target objects, so as to obtain an AR effect that combines virtuality and reality and matches the specific application. Exemplarily, the target object may involve faces, limbs, gestures, actions, etc. related to the human body, or markers and markers related to objects, or sand tables, display areas or display items related to venues or places. Vision-related algorithms may involve visual positioning, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, and position or depth detection of objects. Specific applications may not only involve interactive scenes such as guided tours, navigation, explanations, reconstructions, virtual effect overlay displays, etc. related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, body beautification, special effects display, virtual model display and other interactive scenes.

[0101] The convolutional neural network can be used to detect or identify the relevant features, states and attributes of the target object. The above convolutional neural network is a network model obtained by model training based on a deep learning framework.

[0102] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0104] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A positioning method, characterized in that: The method comprises: Comparing the uncertainty of the decentralized transformation relationship between the cloud and the device with a first preset threshold; wherein the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device; In response to the uncertainty of the decentralized transformation relationship being greater than the first preset threshold, updating the decentralized transformation relationship according to current cloud location information and current device-side tracking information matching the current cloud location information; Using the updated decentralized transformation relationship to perform fusion positioning on the subsequent cloud positioning information and the device-side tracking information; Before the step of comparing the uncertainty of the decentralized transformation relationship between the cloud and the device end with a first preset threshold, the method includes: obtaining initial cloud positioning information; initializing the decentralized transformation relationship according to the initial cloud positioning information and initial device end tracking information matching the initial cloud positioning information; The method further includes: obtaining the uncertainty of the initialized decentralized transformation relationship based on the cloud mapping error and the observation error when the initial cloud positioning information is matched with the initial device-side tracking information.

2. The method according to claim 1, characterized in that The method comprises: The uncertainty of the decentralized transformation relationship is updated according to the accumulated error of the device end during operation.

3. The method according to claim 2, characterized in that The updating of the uncertainty of the decentralized transformation relationship according to the accumulated error of the device during operation includes: The uncertainty of the decentralized transformation relationship is updated using the covariance matrix of the posture information detected by the device during operation.

4. The method according to claim 2, characterized in that: The updating of the uncertainty of the decentralized transformation relationship according to the accumulated error of the device during operation includes: The uncertainty of the decentralized transformation relationship is updated at a preset error accumulation rate.

5. The method according to claim 4, characterized in that The preset error accumulation rate is positively correlated with the tracking duration or tracking distance of the device end.

6. The method according to claim 1, characterized in that The map used in the cloud includes several sub-maps, and there is a preset map association relationship between any two sub-maps. The decentralized transformation relationship includes a first transformation relationship between the global coordinate system of the sub-map observed during initialization and the local coordinate system of the device end, and a second transformation relationship between the global coordinate system of the unobserved sub-map calculated through the first transformation relationship and the preset map association relationship and the local coordinate system of the device end.

7. The method according to claim 1, characterized in that Before the step of using the updated decentralized transformation relationship to fuse and locate the subsequent cloud positioning information and the device-side tracking information, the method further includes: Projecting the subsequent cloud positioning information of the cloud onto the local coordinate system of the device end through the decentralized transformation relationship before updating and the decentralized transformation relationship after updating, respectively, to obtain a first reprojection value and a second reprojection value; comparing an error between the first reprojection value and the second reprojection value with a second preset threshold; In response to the error between the first reprojection value and the second reprojection value being greater than the second preset threshold, executing the step of using the updated decentralized transformation relationship to fuse and locate the subsequent cloud positioning information and the device-side tracking information; In response to the error between the first reprojection value and the second reprojection value being no greater than the second preset threshold, the decentralized transformation relationship before the update is used to perform fusion positioning on the subsequent cloud positioning information and the device-side tracking information.

8. The method according to claim 7, characterized in that Before executing the step of using the updated decentralized transformation relationship to fuse and locate the subsequent cloud positioning information and the device-side tracking information, the method further includes: Comparing an error between the first reprojection value and the second reprojection value corresponding to subsequent cloud location information of a plurality of consecutive frames of the cloud with a second preset threshold; In response to the fact that the errors between the first reprojection value and the second reprojection value corresponding to the subsequent cloud positioning information of the cloud for several consecutive frames are greater than the second preset threshold, the decentralized transformation relationship is reinitialized according to the latest subsequent cloud positioning information of the cloud and the device-side tracking information matching the latest subsequent cloud positioning information of the cloud, and the subsequent cloud positioning information and the device-side tracking information are fused and positioned using the reinitialized decentralized transformation relationship; In response to the fact that the errors between the first reprojection value and the second reprojection value corresponding to the subsequent cloud positioning information of the cloud for several consecutive frames are both greater than the second preset threshold, the step of using the updated decentralized transformation relationship to fuse and locate the subsequent cloud positioning information and the device-side tracking information is executed.

9. The method according to claim 7 or 8, characterized in that: The device end has at least two different operating stages, wherein the second preset thresholds in different operating stages are different.

10. A positioning device, characterized in that: include: A judgment module, used to compare the uncertainty of the decentralized transformation relationship between the cloud and the device with a first preset threshold; wherein the decentralized transformation relationship is used to characterize the coordinate transformation relationship between the global coordinate system of the cloud and the local coordinate system of the device; an updating module, configured to update the decentralized transformation relationship according to current cloud location information and current device-side tracking information matching the current cloud location information, in response to the uncertainty of the decentralized transformation relationship being greater than the first preset threshold; A positioning module, used to use the updated decentralized transformation relationship to perform fusion positioning on the subsequent cloud positioning information and the device-side tracking information; An initialization module is used to perform the following steps before the step of comparing the uncertainty of the decentralized transformation relationship between the cloud and the device with a first preset threshold: obtaining initial cloud positioning information; initializing the decentralized transformation relationship according to the initial cloud positioning information and the initial device tracking information matching the initial cloud positioning information; the initialization module is also used to obtain the uncertainty of the initialized decentralized transformation relationship based on the cloud mapping error and the observation error when the initial cloud positioning information is matched with the initial device tracking information.

11. A positioning device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the positioning method according to any one of claims 1 to 9.

12. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by the processor, the end-cloud fusion positioning described in any one of claims 1 to 9 is implemented.

Images