Positioning method and apparatus for terminal device, electronic device and medium

Abstract

Provided in the present disclosure are a positioning method and apparatus for a terminal device, an electronic device, and a medium. A specific implementation solution comprises: acquiring a first image collected by a terminal device, a plurality of spatial anchors, and the association relationship between the plurality of spatial anchors; on the basis of the first image and the plurality of spatial anchors, determining the pose of the terminal device with respect to each spatial anchor; and, on the basis of the pose of the terminal device with respect to at least one spatial anchor among the plurality of spatial anchors and the association relationship between the plurality of spatial anchors, determining the pose of the terminal device with respect to the other spatial anchors among the plurality of spatial anchors.

Description

Positioning method and device for terminal equipment, electronic equipment and medium

[0001] The present disclosure claims priority to the Chinese patent application No. CN202410852870.5, filed on June 27, 2024, and entitled "Positioning method and device for terminal equipment, electronic equipment and medium", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to the technical field of positioning, and in particular to a positioning method and device for terminal equipment, electronic equipment and medium. BACKGROUND

[0003] In an application scenario of augmented reality, mixed reality, etc., a virtual image can be associated with a spatial anchor, so as to display the virtual image to the terminal equipment for a user to view when the user views the position of the spatial anchor through the terminal equipment. SUMMARY

[0004] The present disclosure aims to provide a positioning method and device for terminal equipment, electronic equipment and medium.

[0005] In a first aspect, the embodiments of the present disclosure provide a positioning method for terminal equipment, comprising: acquiring a first image collected by a terminal equipment, a plurality of spatial anchors and an association relationship between the plurality of spatial anchors; determining a pose of the terminal equipment relative to each spatial anchor based on the first image and the plurality of spatial anchors; and determining a pose of the terminal equipment relative to other spatial anchors in the plurality of spatial anchors based on the pose of the terminal equipment relative to at least one spatial anchor in the plurality of spatial anchors and the association relationship between the plurality of spatial anchors.

[0006] In a second aspect, the embodiments of the present disclosure provide a positioning device for terminal equipment, comprising: an acquisition module configured to acquire a first image collected by a terminal equipment, a plurality of spatial anchors and an association relationship between the plurality of spatial anchors; a first processing module configured to determine a pose of the terminal equipment relative to each spatial anchor based on the first image and the plurality of spatial anchors; and a second processing module configured to determine a pose of the terminal equipment relative to other spatial anchors in the plurality of spatial anchors based on the pose of the terminal equipment relative to at least one spatial anchor in the plurality of spatial anchors and the association relationship between the plurality of spatial anchors.

[0007] In a third aspect, the embodiments of the present disclosure provide a computer-readable storage medium, which stores a computer program for executing the positioning method for terminal equipment provided by any of the embodiments of the present disclosure.

[0008] In a fourth aspect, an electronic device is provided, and the electronic device includes a processor, a memory for storing processor-executable instructions, and the processor is configured to read the executable instructions from the memory and execute the instructions to implement the positioning method for a terminal device provided in any of the embodiments of the present disclosure.

[0009] In a fifth aspect, a head-mounted display device is provided, and the head-mounted display device includes a processor, a memory for storing processor-executable instructions, and the processor is configured to read the executable instructions from the memory and execute the instructions to implement the positioning method for a terminal device provided in any of the embodiments of the present disclosure.

[0010] In a fifth aspect, a head-mounted display device is provided, and the head-mounted display device includes a processor, a memory for storing processor-executable instructions, and the processor is configured to read the executable instructions from the memory and execute the instructions to implement the positioning method for a terminal device provided in any of the embodiments of the present disclosure.

[0011] In a sixth aspect, a computer program product is provided, and the computer program product includes computer program instructions, and the computer program instructions, when executed by a processor, cause the processor to perform the positioning method for a terminal device provided in any of the embodiments of the present disclosure.

[0012] The technical solutions of the present disclosure will be described in further detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0013] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters indicate like features throughout and wherein:

[0014] FIG. 1 is an exemplary system architecture that can be applied to embodiments of the positioning method or apparatus for a terminal device of the present disclosure;

[0015] FIG. 2 is a flowchart of a positioning method for a terminal device provided in an exemplary embodiment of the present disclosure;

[0016] FIG. 3 is a flowchart of determining a pose of a terminal device relative to other spatial anchors in a plurality of spatial anchors provided in an exemplary embodiment of the present disclosure;

[0017] FIG. 4 is a flowchart of a positioning method for a terminal device provided in another exemplary embodiment of the present disclosure;

[0018] FIG. 5 is a flowchart of determining an association relationship between a plurality of spatial anchors provided in an exemplary embodiment of the present disclosure;

[0019] FIG. 6 is a flowchart of determining an association relationship between a plurality of spatial anchors provided in another exemplary embodiment of the present disclosure;

[0020] FIG. 7 is a flow diagram of a positioning method for a terminal device according to another example embodiment of the present disclosure;

[0021] FIG. 8 is a flow diagram of a positioning method for a terminal device according to yet another example embodiment of the present disclosure;

[0022] FIG. 9 is a flow chart of a positioning method for a terminal device according to an example embodiment of the present disclosure;

[0023] FIG. 10 is a schematic diagram of a spatial anchor according to an example embodiment of the present disclosure;

[0024] FIG. 11 is a schematic diagram of a positioning apparatus for a terminal device according to an example embodiment of the present disclosure;

[0025] FIG. 12 is a schematic diagram of a positioning apparatus for a terminal device according to another example embodiment of the present disclosure;

[0026] FIG. 13 is a schematic diagram of a positioning apparatus for a terminal device according to yet another example embodiment of the present disclosure;

[0027] FIG. 14 is a schematic diagram of an electronic device according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In the following, example embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, and the present disclosure is not limited to the described example embodiments.

[0029] It should be noted that the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0030] Those skilled in the art can understand that the terms "first", "second", and the like in the embodiments of the present disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they represent a necessary logical sequence between them.

[0031] It should also be understood that in the embodiments of the present disclosure, "a plurality of" can mean two or more, and "at least one" can mean one, two, or more.

[0032] It should also be understood that the description of the various embodiments of the present disclosure focuses on the differences between the various embodiments, and the same or similar parts can be referred to each other, and for the sake of brevity, will not be repeated.

[0033] The embodiments of the present disclosure can be applied to electronic devices such as head-mounted display devices, terminal devices, servers, and the like, which can operate with many other general-purpose or special-purpose computing system environments or configurations. The electronic devices such as head-mounted display devices, terminal devices, servers, and the like can be described in the general context of computer system executable instructions such as program modules being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like, which perform particular tasks or implement particular abstract data types. The computer system / server can be implemented in a distributed cloud computing environment, in which tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in local or remote computer system storage media, including storage devices.

[0034] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to illustrate and explain the present disclosure, and are not intended to limit the present disclosure.

[0035] FIG. 1 is an exemplary system architecture to which embodiments of a positioning method or apparatus for a terminal device of the present disclosure can be applied. As shown in FIG. 1, the system architecture can include a terminal device 1, a network 2, and an electronic device 3. The network 2 can provide a medium for a communication link between the terminal device 1 and the electronic device 3. The network 2 can include various connection types, such as wired, wireless communication links, or fiber optic cables, and the like. The terminal device 1 is a terminal device to be positioned. The terminal device 1 can be a head-mounted display device, a mobile phone, a tablet, or the like, which is an electronic device having an image capturing function, an image display function, and an audio playback function.

[0036] A user can view a planar or stereoscopic picture or video through a head-mounted display device. The head-mounted display device includes, but is not limited to, an augmented reality (AR) device, a virtual reality (VR) device, a mixed reality (MR) device, and the like.

[0037] The head-mounted display device can generally include an optical imaging system, an acoustic system, and a frame that carries the optical imaging system and the acoustic system. Through the optical imaging system, a user can view an appropriately sized image. Through the frame, the user can wear the head-mounted display device on the head to view the image through the optical imaging system. The frame can be in the form of glasses, a headband, a helmet, or the like.

[0038] The head-mounted display device can be an all-in-one device, and the all-in-one head-mounted display device integrates the functions of a computing unit. The computing unit can provide processing capabilities for the head-mounted display device, and various client applications installed on the head-mounted display device are actually installed on the computing unit. Alternatively, the head-mounted display device can also be a split device, and the split head-mounted display device includes a head-mounted display and a computing unit arranged separately from the head-mounted display. The display screen of the head-mounted display is used to display the image processed by the computing unit. The computing unit in the split head-mounted display device can be a terminal device, which can be a mobile phone, a handle, an adapter, a PC, a tablet, a car machine, etc. For the head-mounted display device as the execution subject, those skilled in the art should understand that for the split head-mounted display device, the execution subject is the computing unit.

[0039] The electronic device 3 can be a mobile phone, a tablet, a computer, a server, etc. The electronic device 3 can provide various services. For example, the electronic device 3 can obtain the first image collected by the terminal device, the plurality of spatial anchors, and the association relationship between the plurality of spatial anchors. Based on the first image and the plurality of spatial anchors, the pose of the terminal device relative to each spatial anchor is determined. Based on the pose of the terminal device relative to at least one of the plurality of spatial anchors and the association relationship between the plurality of spatial anchors, the pose of the terminal device relative to other spatial anchors in the plurality of spatial anchors is determined. Accordingly, the positioning apparatus for the terminal device can also be arranged in the electronic device 3.

[0040] It should be pointed out that the positioning method for the terminal device provided by the present disclosure can also be applied to the terminal device 1. In the case that the method of the present disclosure is applied to the terminal device 1, the terminal device 1 can obtain the first image collected by the terminal device 1, the plurality of spatial anchors, and the association relationship between the plurality of spatial anchors. Based on the first image and the plurality of spatial anchors, the pose of the terminal device 1 relative to each spatial anchor is determined. Based on the pose of the terminal device 1 relative to at least one of the plurality of spatial anchors and the association relationship between the plurality of spatial anchors, the pose of the terminal device 1 relative to other spatial anchors in the plurality of spatial anchors is determined. Accordingly, the positioning apparatus for the terminal device can also be arranged in the terminal device 1.

[0041] In addition to the terminal device 1 and the electronic device 3, the positioning method for the terminal device can also be applied to a server, which can be a background server.

[0042] Exemplary method

[0043] FIG. 2 is a flowchart of a positioning method for a terminal device provided by an exemplary embodiment of the present disclosure. The embodiments of the present disclosure can be applied to an electronic device, as shown in FIG. 2, and the method can include the following steps:

[0044] At step 210, the first image collected by the terminal device, the plurality of spatial anchors, and the association relationship between the plurality of spatial anchors are obtained.

[0045] In some optional embodiments of the present disclosure, the terminal device can be any device with image collection function, for example, the terminal device can be a head-mounted display device, a mobile phone, a tablet, or the like.

[0046] In some optional embodiments of the present disclosure, a spatial anchor refers to a local visual map in a space, and the spatial anchor can be represented as a point cloud including a set of physical coordinate points corresponding to a space region. The spatial anchor can be used for positioning the terminal device, i.e., determining the pose of the terminal device relative to the spatial anchor, to achieve display control of the terminal device and other purposes. For example, a plurality of spatial anchors can be generated in a plurality of regions in a fixed space, respectively, for positioning the terminal device, so as to display corresponding virtual images, virtual objects, or the like at the terminal device when the terminal device camera can see the spatial anchor.

[0047] In some optional embodiments of the present disclosure, the association relationship between the plurality of spatial anchors can include a coordinate system transformation relationship between each two spatial anchors. A corresponding local coordinate system can be set for each spatial anchor, and the association relationship between the plurality of spatial anchors can be established.

[0048] In some optional embodiments of the present disclosure, when the terminal device works in a space where a plurality of spatial anchors are located, the terminal device can obtain a collected image (referred to as a first image) in real time through an image collection device (for example, a camera on the terminal device) on the terminal device, and obtain the plurality of spatial anchors and the association relationship between the plurality of spatial anchors.

[0049] In some optional embodiments of the present disclosure, the association relationship between the plurality of spatial anchors can be pre-established and stored.

[0050] In some optional embodiments of the present disclosure, the association relationship between the plurality of spatial anchors can be determined in real time according to the positioning condition during the movement of the terminal device. For example, at a time t, the terminal device can simultaneously observe two spatial anchors A and B, and the association relationship between A and B can be determined according to the point cloud of A and B. For another example, at a time t, the terminal device observes a spatial anchor A, and at a time r, the terminal device observes a spatial anchor B, and the association relationship between A and B can be established according to the pose of the terminal device relative to the spatial anchor A at the time t and the pose of the terminal device relative to the spatial anchor B at the time r.

[0051] The execution subject (which can be an electronic device, for example, a head-mounted display device, a mobile phone, a server, or the like) of the positioning method for the terminal device obtains the first image collected by the terminal device, the plurality of spatial anchors, and the association relationship between the plurality of spatial anchors.

[0052] In step 220, a pose of the terminal device relative to each spatial anchor is determined based on the first image and the plurality of spatial anchors.

[0053] In some optional embodiments of the present disclosure, the first image represents an observation of the terminal device to the spatial environment, and the pose of the terminal device relative to each spatial anchor can be located through a matching result of the first image and the plurality of spatial anchors. In the embodiments of the present disclosure, the pose of the terminal device relative to each spatial anchor determined based on the image and the spatial anchor can be referred to as visual positioning, visual positioning of the spatial anchor, etc. For any spatial anchor, the pose of the terminal device relative to the spatial anchor can be located through a two-dimensional point on the first image and a three-dimensional point of the spatial anchor. For example, the pose of the terminal device relative to the spatial anchor can be obtained by, but not limited to, a pnp algorithm. If the first image and the spatial anchor fail to be solved, it indicates that the pose of the terminal device relative to the spatial anchor cannot be obtained, i.e., the visual positioning fails, and the pose of the terminal device relative to the spatial anchor can be information indicating that the visual positioning fails, i.e., an invalid pose. If the first image and the spatial anchor are successfully solved, it indicates that the visual positioning succeeds, and a valid pose of the terminal device relative to the spatial anchor can be obtained. That is, at any moment, the visual positioning result of the terminal device relative to any spatial anchor based on the first image collected by the terminal device can include two states of successful positioning and failed positioning.

[0054] In some optional embodiments of the present disclosure, the pose can include a position and an attitude, and can be represented as a translation vector and a rotation matrix, or as an overall transformation matrix including translation and rotation. For example, the pose of the terminal device relative to the spatial anchor can include a translation vector and a rotation matrix of a coordinate system of the terminal device relative to a coordinate system of the spatial anchor.

[0055] In step 230, a pose of the terminal device relative to other spatial anchors in the plurality of spatial anchors is determined based on the pose of the terminal device relative to at least one spatial anchor in the plurality of spatial anchors and a correlation relationship between the plurality of spatial anchors.

[0056] In some optional embodiments of the present disclosure, the other spatial anchors in the plurality of spatial anchors refer to spatial anchors other than the at least one spatial anchor in the plurality of spatial anchors. For example, the plurality of spatial anchors include four spatial anchors A, B, C and D, and the at least one spatial anchor in the plurality of spatial anchors includes two spatial anchors A and B, and the other spatial anchors are two spatial anchors C and D. For example, the pose of the terminal device relative to the two spatial anchors C and D in the plurality of spatial anchors can be determined based on the pose of the terminal device relative to A and B and the correlation relationship between the plurality of spatial anchors.

[0057] In some optional embodiments of the present disclosure, in the case that the pose of the terminal device relative to at least one spatial anchor in the plurality of spatial anchors can be obtained, the pose of the terminal device relative to other spatial anchors in the plurality of spatial anchors can be further obtained in combination with the correlation relationship between the plurality of spatial anchors.

[0058] The positioning method for a terminal device provided by the embodiments of the present disclosure can assist the terminal device in positioning relative to each spatial anchor through the association relationship between multiple spatial anchors when it is necessary to position the pose of the terminal device relative to a spatial anchor. For example, in the case of visual positioning failure of a part of the spatial anchors (i.e., the pose of the terminal device relative to a part of the spatial anchors cannot be obtained through visual images), the terminal device positioning of the part of the spatial anchors with visual positioning failure (i.e., obtaining the pose of the terminal device relative to the part of the spatial anchors) can be completed in combination with the association relationship between the spatial anchors and the pose of the terminal device relative to the spatial anchors with successful visual positioning (i.e., the pose of the terminal device relative to the spatial anchors), thereby improving the positioning success rate of the terminal device.

[0059] FIG. 3 is a flowchart of determining the pose of a terminal device relative to other spatial anchors in a plurality of spatial anchors according to an example embodiment of the present disclosure.

[0060] In some optional embodiments of the present disclosure, as shown in FIG. 3, the step 230 of determining the pose of the terminal device relative to other spatial anchors in the plurality of spatial anchors based on the pose of the terminal device relative to at least one spatial anchor in the plurality of spatial anchors and the association relationship between the plurality of spatial anchors can include:

[0061] The step 2310 can include determining whether the pose of the terminal device relative to each spatial anchor is successfully positioned.

[0062] In some optional embodiments of the present disclosure, for any spatial anchor, in the case of visual positioning of the terminal device based on the first image and the spatial anchor, the visual positioning result can include two states of positioning success and positioning failure. For example, the spatial anchor is not within the observation angle range of the terminal device or the spatial anchor is not within the optimal angle range of the terminal device, resulting in that the two-dimensional feature points in the first image and the visual map of the spatial anchor cannot be matched to obtain enough valid point pairs, so that the pose of the terminal device relative to the spatial anchor cannot be solved, which is likely to cause the visual positioning failure of the terminal device relative to the spatial anchor. Based on this, the state of whether the pose of the terminal device relative to each spatial anchor is successfully positioned can be obtained according to the visual positioning of each spatial anchor.

[0063] In some optional embodiments of the present disclosure, if the visual positioning results corresponding to each spatial anchor are all positioning failure, i.e., the pose of the terminal device relative to each spatial anchor is all positioning failure, it indicates that the positioning of the terminal device at the current moment fails, and the image can be reacquired for positioning at the next moment until the pose of the terminal device relative to at least one spatial anchor in the plurality of spatial anchors is obtained.

[0064] In step 2320, a first spatial anchor corresponding to a successful positioning is determined, and a pose of the terminal device relative to the first spatial anchor is determined as a pose of the terminal device relative to at least one spatial anchor in the plurality of spatial anchors.

[0065] In some optional embodiments of the present disclosure, the spatial anchor in the plurality of spatial anchors for which the visual positioning is successful is referred to as a first spatial anchor, and the number of the first spatial anchors can be at least one. The pose of the terminal device relative to the first spatial anchor is determined as a pose of the terminal device relative to at least one spatial anchor in the plurality of spatial anchors, for the positioning of the pose of the terminal device relative to other spatial anchors. For example, the first spatial anchors include spatial anchors A and B, and the pose of the terminal device relative to at least one spatial anchor in the plurality of spatial anchors includes a pose of the terminal device relative to the spatial anchor A and a pose of the terminal device relative to the spatial anchor B.

[0066] In step 2330, based on the pose of the terminal device relative to the first spatial anchor, and an association relationship between other spatial anchors in the plurality of spatial anchors and the first spatial anchor, a pose of the terminal device relative to the other spatial anchors is determined.

[0067] In some optional embodiments of the present disclosure, the other spatial anchors are spatial anchors in the plurality of spatial anchors other than the first spatial anchors. For example, the other spatial anchors include spatial anchors C and D. Based on the pose of the terminal device relative to the spatial anchors A and B, and the association relationship between the spatial anchors C and D and the spatial anchors A and B, the pose of the terminal device relative to the spatial anchors C and D is determined.

[0068] In the embodiments of the present disclosure, by determining the first spatial anchor in the plurality of spatial anchors for which the visual positioning is successful, an effective reference is provided for the positioning of other spatial anchors for which the visual positioning fails, and the positioning of the terminal device relative to the other spatial anchors is facilitated, thereby effectively improving the success rate of positioning.

[0069] In some optional embodiments of the present disclosure, the positioning method for the terminal device can further include: acquiring an image collected by the terminal device and a plurality of spatial anchors; based on the image and the plurality of spatial anchors, solving a pose of the terminal device relative to each spatial anchor; determining a first spatial anchor in the plurality of spatial anchors for which the visual positioning is successful; acquiring an association relationship between the first spatial anchor and a spatial anchor for which the visual positioning fails; and based on the pose of the terminal device relative to the first spatial anchor and the association relationship, determining a pose of the terminal device relative to the spatial anchor for which the visual positioning fails.

[0070] In some optional embodiments of the present disclosure, based on the pose of the terminal device relative to the first spatial anchor, and an association relationship between other spatial anchors in the plurality of spatial anchors and the first spatial anchor, a pose of the terminal device relative to the other spatial anchors is determined, including:

[0071] For any other spatial anchor, in response to the relative transformation relationship between the other spatial anchor and the first spatial anchor being obtained in the preset manner, the pose of the terminal device relative to the first spatial anchor is transformed according to the relative transformation relationship, and the pose of the terminal device relative to the other spatial anchor is obtained.

[0072] In some optional embodiments of the present disclosure, the preset manner is a point cloud matching calculation manner. For example, if two spatial anchors can be observed in the same visual range, the relative transformation relationship between the two spatial anchors can be accurately calculated through the matching relationship between the three-dimensional points of the two spatial anchors.

[0073] In some optional embodiments of the present disclosure, the first spatial anchor is denoted as A, and for any other spatial anchor C, if there is a relative transformation relationship between the other spatial anchor C and the first spatial anchor A calculated through point cloud matching, the pose of the terminal device relative to the first spatial anchor A can be transformed according to the relative transformation relationship, and the pose of the terminal device relative to the other spatial anchor C is obtained.

[0074] In the case where the relative transformation relationship between the other spatial anchor and the first spatial anchor is obtained through point cloud matching, the present embodiment of the present disclosure has the following advantages. Since the relative transformation relationship can accurately represent the transformation relationship between the other spatial anchor and the first spatial anchor, the pose of the terminal device relative to the other spatial anchor obtained through transformation has high accuracy. The above result can be used as the pose of the terminal device relative to the other spatial anchor, thereby realizing accurate positioning of the terminal device relative to the other spatial anchor in the case where the other spatial anchor fails in visual positioning, and ensuring the accuracy of the positioning result on the basis of improving the positioning success rate.

[0075] FIG. 4 is a flowchart of a positioning method for a terminal device according to another example embodiment of the present disclosure.

[0076] In some optional embodiments of the present disclosure, as shown in FIG. 4, the method of the present embodiment of the present disclosure can further include:

[0077] In step 310, the association relationship between the plurality of spatial anchors is determined.

[0078] In some optional embodiments of the present disclosure, the association relationship between the plurality of spatial anchors can be established in advance through the local coordinate systems of the plurality of spatial anchors.

[0079] In some optional embodiments of the present disclosure, the association relationship between the plurality of spatial anchors can be established based on the pose of the terminal device relative to the spatial anchor at a historical moment.

[0080] According to the embodiments of the present disclosure, the association relationship between the plurality of spatial anchors can be determined, and the association relationship between the plurality of spatial anchors can be used to assist the positioning of a single spatial anchor at any subsequent time, thereby laying an important foundation for improving the subsequent positioning success rate.

[0081] FIG. 5 is a flowchart of determining the association relationship between the plurality of spatial anchors according to an exemplary embodiment of the present disclosure.

[0082] In some optional embodiments of the present disclosure, as shown in FIG. 5, determining the association relationship between the plurality of spatial anchors can include the following steps:

[0083] In step 3110, for any second spatial anchor and third spatial anchor in the plurality of spatial anchors, a first pose of the terminal device relative to the second spatial anchor at a first time and a second pose of the terminal device relative to the third spatial anchor at the first time are obtained.

[0084] In some optional embodiments of the present disclosure, the first time can be any time before the current time.

[0085] In some optional embodiments of the present disclosure, the first time can be the time closest to the current time at which the visual positioning of the second spatial anchor and the third spatial anchor can be completed at the same time.

[0086] In some optional embodiments of the present disclosure, the first pose of the terminal device relative to the second spatial anchor at the first time can be obtained based on the visual positioning of the second spatial anchor based on the image collected by the terminal device at the first time. Similarly, the second pose of the terminal device relative to the third spatial anchor at the first time can be obtained based on the visual positioning of the third spatial anchor based on the image collected by the terminal device at the first time.

[0087] In some optional embodiments of the present disclosure, if the first pose of the terminal device relative to the second spatial anchor at the first time and the second pose of the terminal device relative to the third spatial anchor at the first time can be obtained, since the field of view of the terminal device does not change, that is, the same image contains both the two-dimensional feature points corresponding to the second spatial anchor and the two-dimensional feature points corresponding to the third spatial anchor, it can be determined that the second spatial anchor and the third spatial anchor exist in the same visible area, that is, there are points in the second spatial anchor and the third spatial anchor that are adjacent to the same spatial point, which can be used to establish the association relationship between the second spatial anchor and the third spatial anchor.

[0088] In step 3120, the association relationship between the second spatial anchor and the third spatial anchor is determined based on the first pose and the second pose.

[0089] In some optional embodiments of the present disclosure, the first pose represents the association relationship of the terminal device with the coordinate system of the second spatial anchor, the second pose represents the association relationship of the terminal device with the coordinate system of the third spatial anchor, and based on the actual pose of the terminal device being unchanged at the same time, the first pose and the second pose can be used to establish the association relationship between the second spatial anchor and the third spatial anchor.

[0090] In the embodiments of the present disclosure, the first pose of the terminal device relative to the second spatial anchor and the second pose of the terminal device relative to the third spatial anchor obtained at any first time can be used to determine the association relationship between the second spatial anchor and the third spatial anchor, thereby providing an effective association relationship reference for subsequent joint positioning.

[0091] In some optional embodiments of the present disclosure, at each time, after the visual positioning of each spatial anchor at the time is completed, the association relationship between two spatial anchors that can be successfully visually positioned at the same time at the time can be established in real time, thereby realizing real-time maintenance of the association relationship between the spatial anchors and further improving the accuracy and effectiveness of the association relationship between the spatial anchors.

[0092] In some optional embodiments of the present disclosure, based on the first pose and the second pose, the association relationship between the second spatial anchor and the third spatial anchor can include:

[0093] Based on the first pose and the second pose, a matching relationship between each spatial point of the second spatial anchor and each spatial point of the third spatial anchor is determined, and based on the matching relationship, a transformation relationship between the coordinate system of the second spatial anchor and the coordinate system of the third spatial anchor is calculated as the association relationship between the second spatial anchor and the third spatial anchor.

[0094] In some optional embodiments of the present disclosure, each spatial point of the second spatial anchor is a coordinate point under the coordinate system of the second spatial anchor, and each spatial point of the third spatial anchor is a coordinate point under the coordinate system of the third spatial anchor. According to the first pose and the second pose, a matching relationship of whether each spatial point of the second spatial anchor and each spatial point of the third spatial anchor belong to the same spatial point can be calculated, and based on this, a matching relationship between each spatial point of the second spatial anchor and each spatial point of the third spatial anchor can be obtained. Based on the matching relationship, a corresponding spatial point pair in the second spatial anchor and the third spatial anchor can be obtained, for example, a spatial point P i in the second spatial anchor corresponds to a spatial point P j in the third spatial anchor, and then based on these spatial point pairs, a transformation relationship (i.e., a relative transformation relationship) between the coordinate system of the second spatial anchor and the coordinate system of the third spatial anchor is calculated as the association relationship between the second spatial anchor and the third spatial anchor.

[0095] In the case of implementing visual positioning of at least two spatial anchors at the same time, the embodiments of the present disclosure can establish an accurate association relationship between the spatial anchors through spatial point matching between the spatial anchors, provide a more accurate association relationship between the spatial anchors for subsequent positioning, and improve the accuracy of subsequent positioning results.

[0096] In some optional embodiments of the present disclosure, FIG. 6 is a flow diagram of determining an association relationship between a plurality of spatial anchors according to another exemplary embodiment of the present disclosure. As shown in FIG. 6, determining an association relationship between a plurality of spatial anchors can include the following steps:

[0097] In step 3130, for any fourth spatial anchor and fifth spatial anchor in the plurality of spatial anchors, a third pose of the terminal device relative to the fourth spatial anchor at a second time, a first global pose of the terminal device at the second time, a fourth pose of the terminal device relative to the fifth spatial anchor at a third time, and a second global pose of the terminal device at the third time are obtained.

[0098] In some optional embodiments of the present disclosure, the second time can be any time when visual positioning is completed. The third time can be a time later than the second time when visual positioning is completed. For example, the second time and the third time can be two historical times before the current time, such as times t-i and t-j, t represents the current time, i and j are positive integers, and i is greater than j. For another example, the second time is a historical time before the current time, such as time t-i, and the third time is the current time t, and i is a positive integer. That is, step 3130 can be a step performed after visual positioning of step 220 is performed at the current time.

[0099] In some optional embodiments of the present disclosure, the third pose of the terminal device relative to the fourth spatial anchor at the second time can be obtained based on visual positioning of an image collected by the terminal device at the second time. The specific positioning manner is similar to the pose of the terminal device relative to the first spatial anchor at the first time, and thus is not described herein. Similarly, the fourth pose of the terminal device relative to the fifth spatial anchor at the third time is not described herein.

[0100] The first global pose of the terminal device at the second time and the second global pose of the terminal device at the third time can be the pose of the terminal device in a reference coordinate system (or a reference coordinate system). The reference coordinate system can be a world coordinate system, a coordinate system with the starting position of the terminal device as the origin, etc. The first global pose and the second global pose can be obtained by positioning of a positioning sensor of the terminal device. The positioning sensor of the terminal device can be determined according to the actual situation of the terminal device, for example, the positioning sensor can include at least one of a camera, an inertial measurement unit (IMU), etc. Taking a head-mounted display device as an example, the global pose of the head-mounted display device can be calculated through data collected by the camera, inertial measurement unit, etc. of the head-mounted display device through a corresponding positioning algorithm, and the specific implementation is not limited.

[0101] In some optional embodiments of the present disclosure, the fourth spatial anchor and the fifth spatial anchor can be any two spatial anchors that meet a preset condition. The preset condition can include that there is no time (i.e., the first time described above) at which the two spatial anchors are simultaneously successfully visually positioned, and the two spatial anchors are successfully visually positioned at two different times. Because if the fourth spatial anchor and the fifth spatial anchor are simultaneously successfully visually positioned at the same first time, the association relationship between the spatial anchors can be established as shown in the embodiment of FIG. 5 described above, to ensure the accuracy of the association relationship. That is, the association relationship between any two spatial anchors can be established in one of the following ways: point cloud matching accurate calculation using the pose at the same time, conversion relationship calculation between the pose at different times and the global pose, etc., according to the visual positioning situation at each time.

[0102] In some optional embodiments of the present disclosure, the association relationship can be updated in real time over time. For example, at time t, the association relationship between the spatial anchor A and the spatial anchor B is determined through the positioning poses at different times, and at any time after time t, if the spatial anchor A and the spatial anchor B are simultaneously visually positioned, the association relationship between the spatial anchor A and the spatial anchor B can be accurately calculated through the spatial point matching relationship between the spatial anchor A and the spatial anchor B, and the accurately calculated association relationship is used to replace the original association relationship for subsequent positioning, so as to improve the accuracy of the pose when the pose of the terminal device relative to the spatial anchor is obtained through the association relationship at the subsequent time.

[0103] In step 3140, the association relationship between the fourth spatial anchor and the fifth spatial anchor is determined based on the third pose, the first global pose, the fourth pose, and the second global pose.

[0104] The first global pose and the second global pose are pose information of the terminal device in a global coordinate system. The global coordinate system and the reference coordinate system described above.

[0105] In some optional embodiments of the present disclosure, the third pose represents a relationship between the terminal device and a coordinate system of the fourth spatial anchor at the second time, and the first global pose represents a relationship between the terminal device and a global coordinate system at the second time. Based on the third pose and the first global pose, a relationship between the coordinate system of the fourth spatial anchor and the global coordinate system can be obtained. Similarly, based on the fourth pose and the second global pose, a relationship between the coordinate system of the fifth spatial anchor and the global coordinate system can be obtained. Based on the global invariance of the global coordinate system, the global coordinate system can be used as a bridge to calculate the relationship between the coordinate system of the fourth spatial anchor and the coordinate system of the fifth spatial anchor.

[0106] In the embodiments of the present disclosure, the relationship between the two spatial anchors can be established by using the global poses of the terminal device at different times as a bridge, in a case where the visual positioning poses of the two spatial anchors are obtained at different times in sequence, so as to facilitate subsequent positioning of the terminal device relative to a single spatial anchor.

[0107] In some optional embodiments of the present disclosure, the positioning method for the terminal device can further include: acquiring an image collected by the terminal device, a first spatial anchor, and a second spatial anchor; based on the image and the plurality of spatial anchors, solving a pose of the terminal device relative to each spatial anchor; based on data collected by a camera and an inertial measurement unit of the terminal device, calculating a global pose of the terminal device in a reference coordinate system; using an effective pose of the terminal device relative to the first spatial anchor and the global pose, establishing a relationship between the first spatial anchor and the second spatial anchor; based on the effective pose of the terminal device relative to the first spatial anchor and the relationship, determining a pose of the terminal device relative to the second spatial anchor. Regarding the first spatial anchor and the second spatial anchor, the effective pose of the terminal device relative to the first spatial anchor can be solved by using the image collected by the terminal device and a point cloud of the first spatial anchor, while the effective pose of the terminal device relative to the second spatial anchor cannot be solved by using the image collected by the terminal device and a point cloud of the second spatial anchor.

[0108] In some optional embodiments of the present disclosure, based on the third pose, the first global pose, the fourth pose, and the second global pose, the relationship between the fourth spatial anchor and the fifth spatial anchor is determined, including:

[0109] Based on the first global pose and the second global pose, a pose change of the terminal device at the third time relative to the second time is calculated; based on the third pose and the pose change of the terminal device at the third time relative to the second time, a pose of the terminal device relative to the fourth spatial anchor at the third time is calculated; based on the pose of the terminal device relative to the fourth spatial anchor at the third time and the fourth pose, a transformation relationship between the coordinate system of the fourth spatial anchor and the coordinate system of the fifth spatial anchor is calculated as the relationship between the fourth spatial anchor and the fifth spatial anchor.

[0110] In some optional embodiments of the present disclosure, the fourth spatial anchor is denoted as anchor A, the fifth spatial anchor is denoted as anchor B, the terminal device is denoted as u, k denotes the second time, r denotes the third time, and M denotes the global coordinate system. The third pose can be expressed as The first global coordinate system can be expressed as The fourth pose can be expressed as The second global coordinate system can be expressed as The transformation relationship between the coordinate system of the fourth spatial anchor and the coordinate system of the fifth spatial anchor can be expressed as follows:

[0111] wherein, denotes the inverse transformation of denotes the inverse transformation of .

[0112] The embodiments of the present disclosure establish the pose change relationship of the terminal device at different times by the global poses of the terminal device at different times, and then obtain the pose of the terminal device relative to the fourth spatial anchor at the third time, i.e., the poses of the terminal device at the third time relative to the fourth spatial anchor and the fifth spatial anchor, respectively, by combining the pose of the terminal device relative to the fourth spatial anchor at the second time and the pose change relationship relative to the terminal device at the second time at the third time. Thus, the terminal device is used as a bridge to effectively establish the correlation between the fourth spatial anchor and the fifth spatial anchor for subsequent positioning.

[0113] FIG. 7 is a flowchart of a positioning method for a terminal device according to another exemplary embodiment of the present disclosure.

[0114] In some optional embodiments of the present disclosure, as shown in FIG. 7, the method of the embodiments of the present disclosure can further include:

[0115] In response to the fact that the pose of the terminal device relative to each spatial anchor cannot be determined based on the first image and the plurality of spatial anchors, step 240 obtains a new image collected by the terminal device as the first image, and repeats the step of determining the pose of the terminal device relative to each spatial anchor based on the first image and the plurality of spatial anchors.

[0116] ​In some optional embodiments of the present disclosure, if the pose of the terminal device relative to each spatial anchor cannot be determined based on the first image and the plurality of spatial anchors, it can be indicated that the pose of the terminal device relative to each spatial anchor in the plurality of spatial anchors cannot be obtained by matching the spatial points of the spatial anchor with the two-dimensional feature points in the first image. For example, the observation angle of the terminal device does not cover any spatial anchor, and therefore there is no two-dimensional feature point in the first image that matches the spatial point of any spatial anchor, so the pose of the terminal device relative to the spatial anchor cannot be obtained by using an algorithm such as pnp. If the pose of the terminal device relative to each spatial anchor cannot be determined based on the first image and the plurality of spatial anchors, it indicates that the terminal device fails to visually locate at the current time corresponding to the first image, and waits for the next time to reacquire an image for positioning. After entering the next time, the next time is the current time, and the new image acquired by the terminal device is used as the first image, and the visual positioning process of the embodiments of the present disclosure is repeated. Based on the first image and the plurality of spatial anchors, the pose of the terminal device relative to each spatial anchor is determined, and a visual positioning result is obtained. If the visual positioning result has a first spatial anchor that succeeds in positioning, the pose of the terminal device relative to the first spatial anchor is obtained, and if there are other spatial anchors that fail in positioning, the pose of the terminal device relative to the other spatial anchors is determined according to the association relationship between the other spatial anchors and the first spatial anchor.

[0117] In some optional embodiments of the present disclosure, at each time, the visual positioning of each spatial anchor in the plurality of spatial anchors can be sequentially performed in a certain spatial anchor order. The sorting principle of the spatial anchor order can be set according to actual needs. For example, the sorting principle of the spatial anchor includes but is not limited to sorting from near to far according to the distance between each spatial anchor and the terminal device, sorting from more to less according to the number of spatial points included by each spatial anchor, random sorting, and the like. For example, the plurality of spatial anchors include five spatial anchors A, B, C, D and E in order, then the first image can be matched with the spatial points of the spatial anchor A to solve the pose of the terminal device relative to A. If the positioning of A is successful, the pose initial value of the terminal device relative to B can be calculated according to the pose of the terminal device relative to A and the association relationship between B and A. Of course, if there is an association relationship between C and A, the pose initial value of the terminal device relative to C can also be calculated. Then the first image is matched with the spatial points of B to solve the pose of the terminal device relative to B. If the positioning of B fails, it can be determined whether the association relationship between B and A is the relative transformation relationship obtained by accurate calculation through point cloud matching. If it is the relative transformation relationship obtained by accurate calculation, the calculated pose initial value of the terminal device relative to B can be used as the alternative positioning result, that is, as the pose of the terminal device relative to B. If the association relationship between B and A is not obtained by accurate calculation through point cloud matching, it can be determined that the positioning of B at the current time fails, or the pose initial value can be used as the pose of the terminal device relative to B, and the confidence of the pose is marked. The specific setting can be made according to actual needs. If the positioning of A fails, the positioning of B is continued, and the process is similar to that of A, which will not be repeated here. Until all spatial anchors are traversed, the positioning results of each spatial anchor are obtained.

[0118] In some optional embodiments of the present disclosure, the visual positioning process of each spatial anchor in the plurality of spatial anchors can be executed in parallel or partially in parallel, so as to improve the processing efficiency. After the visual positioning is completed, each spatial anchor with successful positioning is taken as a target spatial anchor, and the pose of the terminal device relative to other spatial anchors is determined according to the association relationship between the other spatial anchors with failed positioning and the target spatial anchor.

[0119] FIG. 8 is a flowchart of a positioning method for a terminal device according to another exemplary embodiment of the present disclosure.

[0120] In some optional embodiments of the present disclosure, the method of the embodiments of the present disclosure can further include:

[0121] In step 410, image data and positioning data collected at a plurality of spatial positions around the target region are acquired.

[0122] In some optional embodiments of the present disclosure, the target region can be a region in space where a spatial anchor is to be established. For example, a region in a conference room space where a spatial anchor of a virtual display screen is to be displayed, a region in an exhibition space where a spatial anchor of a virtual exhibition item, a virtual image, etc. is to be displayed. The plurality of spatial positions around the target region can refer to positions where the terminal device collecting image data collects images of the target region, and at the plurality of spatial positions, images of different perspectives of the target region can be collected, such as images of different perspectives such as front view, side view, top view, etc. The positioning data of the corresponding terminal device is obtained at each spatial position. The positioning data can include sensor data for positioning the pose of the terminal device in the global coordinate system. The sensor data can include IMU data and other positioning-related data.

[0123] In some optional embodiments of the present disclosure, the terminal device collecting image data and positioning data at the plurality of spatial positions around the target region can be the same device as the terminal device collecting the first image, or can be different devices.

[0124] In some optional embodiments of the present disclosure, the terminal device collecting image data and positioning data at the plurality of spatial positions around the target region can include one or more devices. For example, a plurality of users wear head-mounted display devices and move in the space respectively, and obtain image data and positioning data of the plurality of users looking at the same region at different spatial positions, as image data and positioning data collected at the plurality of spatial positions around the region.

[0125] Step 420, based on the image data and positioning data respectively corresponding to each spatial position, a spatial anchor corresponding to the target region is generated.

[0126] In some optional embodiments of the present disclosure, the image data respectively corresponding to each spatial position can contain feature points under different perspectives of the target region, and in combination with the positioning data of each spatial position, a point cloud of the target region can be established, and the point cloud can be taken as the spatial anchor corresponding to the target region. The established point cloud of the target region can also be densified, and the densified point cloud can be taken as the spatial anchor corresponding to the target region.

[0127] In some optional embodiments of the present disclosure, according to the positioning data of each spatial position, a correlation between the spatial ranges covered by the image data of each spatial position can be established. Through the matching relationship between the two-dimensional feature points of the image data of each spatial position and the spatial points, in combination with the correlation between the spatial ranges of different perspectives, the same spatial points corresponding to the images of different perspectives in space can be determined, and based on this, each spatial point in the target region is obtained, which constitutes the spatial anchor corresponding to the target region. The spatial anchor of the target region can be realized based on any implementable multi-perspective image-based three-dimensional reconstruction technology.

[0128] In some optional embodiments of the present disclosure, the image data and the positioning data of the plurality of target regions can be collected respectively, and the spatial anchors corresponding to the target regions respectively can be established.

[0129] The embodiments of the present disclosure effectively construct the spatial anchors of the target regions by the image data and the positioning data collected at the plurality of spatial positions around the target regions, and provide accurate and effective spatial anchors for the positioning of the terminal device.

[0130] In some optional embodiments of the present disclosure, FIG. 9 is a flow chart of a positioning method for a terminal device provided by an exemplary embodiment of the present disclosure. As shown in FIG. 9, the method of the embodiments of the present disclosure can include the following steps:

[0131] Step 510, a plurality of spatial anchors (i.e., anchors) are generated.

[0132] Step 5210, anchor A is positioned, i.e., the pose of the terminal device relative to anchor A is determined based on the collected first image at each time. Go to step 5220.

[0133] Step 5310, anchor B is positioned, i.e., the pose of the terminal device relative to anchor B is determined based on the collected first image at each time. Go to step 5320.

[0134] Step 5220, it is determined whether anchor A is successfully positioned at time k. If the positioning fails, go to step 5230. If the positioning succeeds, go to step 5240 and step 540. Time k is any time.

[0135] Step 5230, the next anchor is positioned.

[0136] Step 5240, the pose of the terminal device relative to anchor A at time k and the global pose of the terminal device at time k are recorded. Go to step 550.

[0137] Step 5320, it is determined whether anchor B is successfully positioned at time k. If the positioning fails, go to step 5330. If the positioning succeeds, go to step 540.

[0138] Step 5330, it is determined whether anchor B is successfully positioned at time r. If the positioning succeeds, the pose of the terminal device relative to anchor B at time r and the global pose of the terminal device are obtained. Time r is any time after time k.

[0139] Step 540, anchor A and anchor B are visually matched, and the relative transformation relationship between anchor A and anchor B is calculated.

[0140] Step 550, according to the pose of the terminal device relative to anchor B at time r, the global pose of the terminal device at time r, the pose of the terminal device relative to anchor A at time k obtained in step 5240, and the global pose of the terminal device at time k, the relative transformation relationship between anchor A and anchor B is calculated.

[0141] Step 5610, positioning anchor A at the current time. The current time can be any time after time r.

[0142] Step 5620, determining whether anchor A is successfully positioned at the current time. If yes, go to step 5630. If no, go to step 5640.

[0143] Step 5630, calculating the initial pose of the terminal device relative to anchor B. Go to step 5640.

[0144] Step 5640, positioning anchor B.

[0145] Step 5650, determining whether anchor B is successfully positioned. If yes, go to step 5680. If no, go to step 5660.

[0146] If anchor B is successfully positioned, the initial pose of the terminal device relative to other anchors (such as anchor C) can be calculated based on the pose of the terminal device relative to anchor B and the association relationship between other anchors (such as anchor C) and anchor B, which is used for subsequent positioning of the other anchors.

[0147] Step 5660, determining whether there is a relative transformation relationship between anchor A and anchor B for visual map matching calculation (i.e. point cloud matching accurate calculation). If yes, go to step 5670. If no, go to step 5690.

[0148] Step 5670, taking the initial pose of the terminal device relative to anchor B calculated in step 5630 as the positioning result of anchor B, that is, taking the initial pose as the pose of the terminal device relative to anchor B.

[0149] Since the relative transformation relationship calculated based on the pose of the terminal device relative to anchor A at time k and the pose of the terminal device relative to anchor B at time r, combined with the global poses at the two times, is easily affected by the drift of the global pose over time, resulting in a large error in the relative transformation relationship, therefore, for the case of visual map matching calculation of the relative transformation relationship, the initial pose is taken as the alternative positioning result, which can ensure the accuracy of the pose.

[0150] Step 5680, anchorB positioning succeeds, and the next anchor is positioned.

[0151] Step 5690, anchorB positioning fails, and the next anchor is positioned.

[0152] It should be noted that the embodiment shown in FIG. 9 is only used as an example to illustrate the positioning process of anchorA and anchorB. For each anchor in multiple anchors, after the visual positioning of the anchor succeeds, the initial pose of the terminal device relative to other anchors can be calculated based on the determined association relationship between the other anchors and the anchor. The positioning process of the other anchors is similar to that of anchorB, and thus is not described herein.

[0153] In some optional embodiments of the present disclosure, FIG. 10 is a schematic diagram of spatial anchors according to an example embodiment of the present disclosure. As shown in FIG. 10, three spatial anchors, namely anchor1, anchor2 and anchor3, are arranged in a space. A virtual image of anchor1 positioning is represented by a five-pointed star, that is, when a user observes anchor1 through a terminal device, a virtual five-pointed star image is displayed on the terminal device. A virtual image of anchor2 positioning is represented by a triangle, and a virtual image of anchor3 positioning is represented by a cylinder. If, at the current time, anchor1 is the best spatial anchor in the user's visual angle, the pose of the terminal device relative to anchor1 can be obtained based on the first image and the point cloud of anchor1. Anchor2 and anchor3 are not in the best visual angle range of the user, and thus the pose of the terminal device relative to anchor2 and the pose of the terminal device relative to anchor3 can be positioned based on the association relationship between anchor2, anchor3 and anchor1. As can be seen, after the positioning result of the spatial anchor with the best visual angle is obtained at any time, the spatial anchors that are not in the best visual angle range and are not in the visual angle range can be positioned based on the association relationship between the spatial anchors, thereby effectively improving the positioning success rate.

[0154] In the related art, some products bind multiple spatial anchors in the same map, and the positioning range of the spatial anchors is easily limited by the size of the single map. Some other products implement local mapping of the space by taking each spatial anchor as a small map. However, due to the small coverage range of the local map, the positioning success rate of a single spatial anchor decreases. The embodiments of the present disclosure can assist the positioning of a single spatial anchor based on a visual map network formed by multiple spatial anchors without limiting the positioning range of the spatial anchor, thereby effectively improving the positioning success rate of the single spatial anchor.

[0155] The above-mentioned embodiments or optional examples of the present disclosure can be implemented alone or in any combination without conflict, and can be set according to actual needs. The present disclosure is not limited.

[0156] Any positioning method for a terminal device provided by the embodiments of the present disclosure can be executed by any appropriate device with data processing capability, including but not limited to: a terminal device, a server, and the like. Alternatively, any positioning method for a terminal device provided by the embodiments of the present disclosure can be executed by a processor, such as a processor executing any positioning method for a terminal device mentioned by the embodiments of the present disclosure by invoking corresponding instructions stored in a memory. Details are omitted hereinafter.

[0157] Exemplary apparatus

[0158] FIG. 11 is a structural schematic diagram of a positioning apparatus for a terminal device provided by an exemplary embodiment of the present disclosure. The apparatus of this embodiment can be used to implement the corresponding method embodiments of the present disclosure. As shown in FIG. 11, the apparatus includes an acquisition module 610, a first processing module 620, and a second processing module 630.

[0159] The acquisition module 610 can be configured to acquire a first image collected by a terminal device, a plurality of spatial anchors, and an association relationship between the plurality of spatial anchors.

[0160] The first processing module 620 can be configured to determine a pose of the terminal device relative to each spatial anchor based on the first image and the plurality of spatial anchors.

[0161] The second processing module 630 can be configured to determine a pose of the terminal device relative to other spatial anchors in the plurality of spatial anchors based on the pose of the terminal device relative to at least one spatial anchor in the plurality of spatial anchors and the association relationship between the plurality of spatial anchors.

[0162] In some optional embodiments of the present disclosure, the second processing module 630 can be specifically configured to determine a state of whether the pose of the terminal device relative to each spatial anchor is successfully positioned. The first spatial anchor corresponding to the successfully positioned pose is determined, and the pose of the terminal device relative to the first spatial anchor is determined as the pose of the terminal device relative to at least one spatial anchor in the plurality of spatial anchors. The pose of the terminal device relative to other spatial anchors in the plurality of spatial anchors other than the first spatial anchor is determined based on the pose of the terminal device relative to the first spatial anchor and the association relationship between the other spatial anchors and the first spatial anchor.

[0163] In some optional embodiments of the present disclosure, the second processing module 630 can be specifically configured to:

[0164] For any other spatial anchor, in response to the relative transformation relationship between the other spatial anchor and the first spatial anchor being obtained in the preset manner, the pose of the terminal device relative to the first spatial anchor is transformed according to the relative transformation relationship, and the pose of the terminal device relative to the other spatial anchor is obtained.

[0165] FIG. 12 is a structural schematic diagram of a positioning apparatus for a terminal device according to another example embodiment of the present disclosure. As shown in FIG. 12, the apparatus according to the embodiment of the present disclosure can further include an association relationship determination module 710, which can be configured to determine the association relationship between the plurality of spatial anchors.

[0166] In some optional embodiments of the present disclosure, the association relationship determination module 710 can be specifically configured to: for any second spatial anchor and third spatial anchor in the plurality of spatial anchors, obtain a first pose of the terminal device relative to the second spatial anchor at a first time and a second pose of the terminal device relative to the third spatial anchor at the first time. Based on the first pose and the second pose, determine the association relationship between the second spatial anchor and the third spatial anchor.

[0167] In some optional embodiments of the present disclosure, the association relationship determination module 710 can be specifically configured to: based on the first pose and the second pose, determine a matching relationship between each spatial point of the second spatial anchor and each spatial point of the third spatial anchor; and based on the matching relationship, calculate a transformation relationship between the coordinate system of the second spatial anchor and the coordinate system of the third spatial anchor as the association relationship between the second spatial anchor and the third spatial anchor.

[0168] In some optional embodiments of the present disclosure, the association relationship determination module 710 can be specifically configured to: for any fourth spatial anchor and fifth spatial anchor in the plurality of spatial anchors, obtain a third pose of the terminal device relative to the fourth spatial anchor at a second time, a first global pose of the terminal device at the second time, a fourth pose of the terminal device relative to the fifth spatial anchor at a third time, and a second global pose of the terminal device at the third time. Based on the third pose, the first global pose, the fourth pose, and the second global pose, determine the association relationship between the fourth spatial anchor and the fifth spatial anchor. The first global pose and the second global pose are pose information of the terminal device in a global coordinate system.

[0169] In some optional embodiments of the present disclosure, the association relationship determining module 710 can be specifically configured to: based on the first global pose and the second global pose, calculate pose change information of the terminal device at the third time relative to the second time; based on the third pose and the pose change information of the terminal device at the third time relative to the second time, calculate the pose of the terminal device relative to the fourth spatial anchor at the third time; and based on the pose of the terminal device relative to the fourth spatial anchor at the third time and the fourth pose, calculate the transformation relationship between the coordinate system of the fourth spatial anchor and the coordinate system of the fifth spatial anchor as the association relationship between the fourth spatial anchor and the fifth spatial anchor.

[0170] In some optional embodiments of the present disclosure, the acquiring module 610 is further configured to, in response to being unable to determine the pose of the terminal device relative to each spatial anchor based on the first image and the plurality of spatial anchors, acquire a new image collected by the terminal device as the first image. Then, the first processing module 620 performs the step of determining the pose of the terminal device relative to each spatial anchor based on the first image and the plurality of spatial anchors, and the second processing module 630 performs the step of determining the pose of the terminal device relative to other spatial anchors in the plurality of spatial anchors based on the pose of the terminal device relative to at least one spatial anchor in the plurality of spatial anchors and the association relationship between the plurality of spatial anchors.

[0171] FIG. 13 is a structural schematic diagram of a positioning apparatus for a terminal device according to another example embodiment of the present disclosure. As shown in FIG. 13, the apparatus according to the embodiment of the present disclosure can further include a data acquiring module 810 and a spatial anchor generating module 820.

[0172] The data acquiring module 810 can be configured to acquire image data and positioning data collected at a plurality of spatial positions around a target region.

[0173] The spatial anchor generating module 820 can be configured to generate spatial anchors corresponding to the target region based on the image data and the positioning data corresponding to each spatial position respectively.

[0174] It should be noted that the specific implementation of the positioning apparatus for a terminal device according to the embodiment of the present disclosure is similar to the specific implementation of the positioning method for a terminal device according to the embodiment of the present disclosure. For details, refer to the part of the positioning method for a terminal device. To reduce redundancy, no further description is given here.

[0175] Example electronic device

[0176] The embodiment of the present disclosure further provides an electronic device, which includes a processor and a memory for storing processor-executable instructions.

[0177] The processor is configured to read and execute the executable instructions from the memory to implement the method according to any one of the embodiments of the present disclosure.

[0178] Example head-mounted display device

[0179] The embodiments of the present disclosure further provide an electronic device, comprising: a processor, and a memory for storing processor-executable instructions.

[0180] The processor is configured to read and execute the executable instructions from the memory to implement the method of any of the above embodiments of the present disclosure.

[0181] FIG. 14 is a structural schematic diagram of an application embodiment of the electronic device of the present disclosure. The structure of the head-mounted display device can refer to the structure of the electronic device. In this embodiment, the electronic device 100 comprises one or more processors 110 and a memory 120.

[0182] The processor 110 can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device 100 to perform desired functions.

[0183] The memory 120 can comprise one or more computer program products, which can comprise various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory can comprise, for example, random access memory (RAM), cache memory, and / or the like. The non-volatile memory can comprise, for example, read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage media, and the processor 110 can run the program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, and the like can also be stored in the computer-readable storage media.

[0184] In one example, the electronic device 100 can further comprise an input device 130 and an output device 140, which are interconnected by a bus system and / or other forms of connection mechanism (not shown).

[0185] For example, the input device 130 can be the microphone or microphone array described above, for capturing input signals of a sound source.

[0186] In addition, the input device 130 can further comprise, for example, a keyboard, a mouse, and the like.

[0187] The output device 140 can output various information to the outside, including the determined distance information, direction information, and the like. The output device 140 can comprise, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0188] Of course, for simplicity, only some of the components of the electronic device 100 related to the present disclosure are shown in FIG. 14, and components such as buses, input / output interfaces, and the like are omitted. In addition thereto, the electronic device 100 can further include any other appropriate components according to a specific application.

[0189] Example computer program product and computer readable storage medium

[0190] In addition to the above-mentioned methods and devices, embodiments of the present disclosure can also be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform steps of the methods according to various embodiments of the present disclosure described in the above "Example Method" section of the specification.

[0191] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0192] In addition, embodiments of the present disclosure can also be a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform steps of the methods according to various embodiments of the present disclosure described in the above "Example Method" section of the specification.

[0193] The computer readable storage medium can be any combination of one or more non-transitory media. The non-transitory medium can be a non-transitory signal medium or a non-transitory storage medium. The non-transitory storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the non-transitory storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0194] It is also important to note that the devices, apparatuses and methods described in the disclosure can be embodied in a variety of other forms. In addition, each of the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalents of the disclosure.

[0195] The above description has been presented for the purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the disclosure to forms disclosed herein. Although several example aspects and embodiments have been discussed above, those of ordinary skill in the art will recognize certain variations, modifications, changes, additions, and sub-combinations thereof.

Claims

1. A positioning method for a terminal device, comprising: Acquire images captured by the terminal device, multiple spatial anchors, and the relationships between the multiple spatial anchors; Based on the image and the plurality of spatial anchors, the pose of the terminal device relative to each spatial anchor is determined; Based on the pose of the terminal device relative to at least one of the plurality of spatial anchors and the association relationship between the plurality of spatial anchors, the pose of the terminal device relative to the other spatial anchors among the plurality of spatial anchors is determined.

2. The method according to claim 1, wherein, Determining the pose of the terminal device relative to the other spatial anchors among the plurality of spatial anchors based on the pose of the terminal device relative to at least one of the plurality of spatial anchors and the association relationships among the plurality of spatial anchors includes: Determine whether the terminal device has been successfully positioned relative to each spatial anchor. The first spatial anchor corresponding to the successfully positioned pose is determined, and the pose of the terminal device relative to the first spatial anchor is determined as the pose of the terminal device relative to at least one of the plurality of spatial anchors. Based on the pose of the terminal device relative to the first spatial anchor and the association between the other spatial anchors (excluding the first spatial anchor) and the first spatial anchor, the pose of the terminal device relative to the other spatial anchors is determined.

3. The method according to claim 1, wherein, Determining the pose of the terminal device relative to each spatial anchor based on the image and the plurality of spatial anchors includes: The pose of the terminal device relative to the spatial anchor is determined by comparing two-dimensional points on the image with three-dimensional points on the spatial anchor. Determining the pose of the terminal device relative to the other spatial anchors among the plurality of spatial anchors based on the pose of the terminal device relative to at least one of the plurality of spatial anchors and the association relationships among the plurality of spatial anchors includes: The effective pose of the terminal device relative to the spatial anchor that has been successfully solved is taken as the pose of the terminal device relative to at least one of the plurality of spatial anchors. The pose of the terminal device relative to other spatial anchors among the multiple spatial anchors is determined by the association between the effective pose and the multiple spatial anchors, wherein the pose of the terminal device relative to the other spatial anchors cannot be solved by using two-dimensional points on the image and three-dimensional points of the other spatial anchors.

4. The method according to any one of claims 1 to 3, further comprising determining the association relationship between the plurality of spatial anchors; wherein determining the association relationship between the plurality of spatial anchors includes: For any second spatial anchor and third spatial anchor among the plurality of spatial anchors, obtain the first pose of the terminal device relative to the second spatial anchor at a first time, and the second pose of the terminal device relative to the third spatial anchor at the first time; Based on the first pose and the second pose, the association between the second spatial anchor and the third spatial anchor is determined.

5. The method according to any one of claims 1 to 3, further comprising determining the association relationship between the plurality of spatial anchors; wherein determining the association relationship between the plurality of spatial anchors includes: For any fourth and fifth spatial anchors among the plurality of spatial anchors, obtain the third pose of the terminal device relative to the fourth spatial anchor at the second time, the first global pose of the terminal device in the reference coordinate system at the second time, the fourth pose of the terminal device relative to the fifth spatial anchor at the third time, and the second global pose of the terminal device in the reference coordinate system at the third time. Based on the third pose, the first global pose, the fourth pose, and the second global pose, the association between the fourth spatial anchor and the fifth spatial anchor is determined; the first global pose and the second global pose are the pose information of the terminal device in the global coordinate system.

6. The method according to claim 5, further comprising: Using the image data captured by the camera of the terminal device, the third pose and the fourth pose are solved. The first global pose and the second global pose are calculated using data collected by the camera and inertial measurement unit of the terminal device.

7. The method according to claim 2, wherein, Determining the pose of the terminal device relative to the other spatial anchors based on the pose of the terminal device relative to the first spatial anchor and the association relationship between the other spatial anchors (excluding the first spatial anchor) and the first spatial anchor includes: For any of the other spatial anchors, in response to the fact that the association between the other spatial anchor and the first spatial anchor is a relative transformation relationship obtained through a preset method, the pose of the terminal device relative to the first spatial anchor is transformed according to the relative transformation relationship to obtain the pose of the terminal device relative to the other spatial anchor.

8. The method according to claim 4, wherein, The step of determining the association between the second spatial anchor and the third spatial anchor based on the first pose and the second pose includes: Based on the first pose and the second pose, determine the matching relationship between each spatial point of the second spatial anchor and each spatial point of the third spatial anchor; Based on the matching relationship, the transformation relationship between the coordinate system of the second spatial anchor and the coordinate system of the third spatial anchor is calculated, wherein the transformation relationship serves as the association relationship between the second spatial anchor and the third spatial anchor.

9. The method according to claim 5, wherein, The step of determining the association between the fourth spatial anchor and the fifth spatial anchor based on the third pose, the first global pose, the fourth pose, and the second global pose includes: Based on the first global pose and the second global pose, calculate the pose change information of the terminal device at the third time relative to the second time. Based on the third pose and the pose change information, calculate the fifth pose of the terminal device relative to the fourth spatial anchor at the third moment; Based on the fifth pose and the fourth pose, the transformation relationship between the coordinate system of the fourth spatial anchor and the coordinate system of the fifth spatial anchor is calculated, wherein the transformation relationship serves as the association relationship between the fourth spatial anchor and the fifth spatial anchor.

10. The method according to any one of claims 1-9, wherein, Also includes: In response to the inability to determine the pose of the terminal device relative to each of the spatial anchors based on the image and the plurality of spatial anchors, a new image acquired by the terminal device is obtained as the image, and the step of determining the pose of the terminal device relative to each of the spatial anchors based on the image and the plurality of spatial anchors is repeated.

11. The method according to any one of claims 1-9, wherein, Also includes: Acquire image and location data from multiple spatial locations surrounding the target area; Based on the image data and positioning data corresponding to each of the aforementioned spatial locations, the spatial anchor corresponding to the target area is generated.

12. The method according to any one of claims 1-9, wherein, Also includes: The virtual image is positioned using the multiple spatial anchors.

13. A computer-readable storage medium storing a computer program for performing the method according to any one of claims 1-12.

14. A head-mounted display device, the head-mounted display device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-12.

15. The head-mounted display device of claim 14, wherein the head-mounted display device is used to display a virtual image positioned by the spatial anchor.

Images