Method and device for spatial positioning

Abstract

The present disclosure provides a method and device for spatial positioning. In the method, a first image of a second device is collected, the first image includes 2D points of the second device and descriptors corresponding to the 2D points; feature point matching is performed on the 2D points of the second device and the 3D points on the 3D model of the second device using the descriptors corresponding to the 3D points on the 3D model of the second device, so as to obtain a first correspondence between at least three non-collinear 2D points of the second device and 3D points of the 3D model of the second device, the 3D model of the second device including the 3D points and the descriptors corresponding to the 3D points; the positioning of the 3D model of the second device in the world coordinate system is determined according to the positioning of the second device in the world coordinate system and the second correspondence between the second device and the 3D model; and the posture of the first device in the world coordinate system is determined according to the positioning of the 3D model of the second device in the world coordinate system and the first correspondence.

Description

Technical Field

[0001] The present disclosure relates to the field of information technology, and in particular, to a method and device for spatial positioning. Background Art

[0002] With the rapid development of various technologies such as computers and communications, technologies such as augmented reality (AR) have also developed rapidly. Specifically, in AR technology, computer technology can be used to apply virtual information to the real world, that is, real scenes and virtual images (for example, virtual objects, virtual scenes, etc.) are superimposed on the same screen in real time, so that the human eye can see a mixed image of real scenes and virtual images at the same time, thereby achieving the effect of augmented reality.

[0003] Users can interact with virtual objects in the AR scene through interactive devices to produce desired effects. For example, users can move the interactive device, and the spatial movement of the interactive device can be converted into the movement of virtual objects in the AR scene, thereby achieving the purpose of controlling the virtual object. Therefore, in the interaction process of the AR scene, it is necessary to determine the spatial positioning of the interactive device. How to accurately determine the positioning of the interactive device in space has become an urgent problem to be solved. Summary of the invention

[0004] In view of the above, the present disclosure provides a method and apparatus for spatial positioning.

[0005] According to one aspect of the present disclosure, a method for spatial positioning is provided, the method being executed by a first device, the method comprising: acquiring a first image of a second device, the first image comprising 2D points of the second device and descriptors corresponding to the 2D points; performing feature point matching on the 2D points of the second device and the 3D points on the 3D model of the second device using descriptors corresponding to the 3D points on the 3D model of the second device and descriptors corresponding to the 2D points, so as to obtain a first correspondence between at least three non-collinear 2D points of the second device and 3D points of the 3D model of the second device, the 3D model of the second device comprising 3D points and descriptors corresponding to the 3D points; determining the positioning of the 3D model of the second device in the world coordinate system according to the positioning of the second device in the world coordinate system and the second correspondence between the second device and the 3D model; and determining the posture of the first device in the world coordinate system according to the positioning of the 3D model of the second device in the world coordinate system and the first correspondence.

[0006] According to another aspect of the present disclosure, there is also provided an apparatus for spatial positioning, which is applied to a first device, and comprises: an image acquisition unit, configured to acquire a first image of a second device, the first image comprising 2D points of the second device and descriptors corresponding to the 2D points; a feature point matching unit, configured to perform feature point matching on the 2D points of the second device and the 3D points on the 3D model of the second device using descriptors corresponding to the 3D points on the 3D model of the second device and descriptors corresponding to the 2D points, so as to obtain a first correspondence between at least three non-collinear 2D points of the second device and 3D points of the 3D model of the second device, the 3D model of the second device comprising 3D points and descriptors corresponding to the 3D points; a positioning unit, configured to determine the positioning of the 3D model of the second device in the world coordinate system according to the positioning of the second device in the world coordinate system and the second correspondence between the second device and the 3D model; and a posture determination unit, configured to determine the posture of the first device in the world coordinate system according to the positioning of the 3D model of the second device in the world coordinate system and the first correspondence.

[0007] According to another aspect of the present disclosure, an electronic device is also provided, including: at least one processor; and a memory, wherein the memory stores instructions, and when the instructions are executed by the at least one processor, the at least one processor executes the method for spatial positioning as described above.

[0008] According to another aspect of the present disclosure, a non-volatile machine-readable storage medium is also provided, which stores executable instructions. When the instructions are executed, the machine performs the method for spatial positioning as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] A further understanding of the nature and advantages of the present disclosure may be achieved by referring to the following drawings.In the accompanying drawings, similar components or features may have the same reference numeral.

[0010] Figure 1 A schematic diagram showing an example of an application scenario of the spatial positioning method according to the present disclosure.

[0011] Figure 2 A flow chart of an example of a spatial positioning method according to the present disclosure is shown.

[0012] Figure 3 A schematic diagram showing an example of a 3D model of a second device according to the present disclosure.

[0013] Figure 4A schematic diagram showing an example of a mapping relationship between 2D points in a first image and 3D points of a 3D model according to the present disclosure.

[0014] Figure 5 A flowchart of another example of the spatial positioning method according to the present disclosure is shown.

[0015] Figure 6 A schematic diagram showing an example relationship among a second device, a 3D model, and a world coordinate system according to the present disclosure is shown.

[0016] Figure 7 A flowchart of another example of the spatial positioning method according to the present disclosure is shown.

[0017] Figure 8 A block diagram of an example of a spatial positioning device according to the present disclosure is shown.

[0018] Fig. 9 A block diagram of another example of a spatial positioning device according to the present disclosure is shown.

[0019] Fig.10 A block diagram of another example of a spatial positioning device according to the present disclosure is shown.

[0020] Fig.11 A block diagram of an electronic device for implementing a spatial positioning method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0021] The subject matter described herein will be discussed below with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and implement the subject matter described herein, and is not a limitation of the scope of protection, applicability or examples set forth in the claims. The functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the present disclosure. Various examples may omit, replace or add various processes or components as needed. In addition, the features described relative to some examples may also be combined in other examples.

[0022] As used herein, the term "including" and its variations represent open terms, meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification.

[0023] In the current application of augmented reality (AR) technology, a terminal device (for example, a mobile terminal such as a mobile phone, a personal computer, a tablet computer, etc.) can be connected to AR glasses, and the terminal device can be used as a handle to control the virtual objects displayed by the AR glasses. The terminal device moves in space to perform corresponding operations on the virtual objects displayed in the AR glasses, such as position movement, posture change, interface switching, selection, cancellation, entry, exit, etc. Based on this, the positioning of the terminal device in space is very important in the AR scene, which directly affects the interaction with the virtual objects in the AR scene.

[0024] At present, terminal devices use built-in IMU (Inertial Measurement Unit) to perform attitude calculation, while general IMUs used in consumer products can only achieve 3DOF (degree of freedom) functions, which only include yaw, pitch and roll. Based on this, terminal devices can only perform attitude calculations through yaw, pitch and roll, which have certain limitations. If only the IMU is used for 6DOF spatial positioning, the spatial positioning of the terminal device will be inaccurate, which will affect the interaction with virtual objects in the AR scene.

[0025] In view of the above, the present disclosure provides a method and device for spatial positioning. In the method, a first image of a second device is collected, the first image includes 2D points of the second device and descriptors corresponding to the 2D points; feature point matching is performed on the 2D points of the second device and the 3D points on the 3D model of the second device using the descriptors corresponding to the 3D points on the 3D model of the second device and the descriptors corresponding to the 2D points, so as to obtain a first correspondence between at least three non-collinear 2D points of the second device and the 3D points of the 3D model of the second device, the 3D model of the second device includes 3D points and descriptors corresponding to the 3D points; according to the positioning of the second device in the world coordinate system and the second correspondence between the second device and the 3D model, the positioning of the 3D model of the second device in the world coordinate system is determined; and according to the positioning of the 3D model of the second device in the world coordinate system and the first correspondence, the posture of the first device in the world coordinate system is determined. Through the technical solution of the present disclosure, the accuracy of spatial positioning of the first device is improved.

[0026] Figure 1 A schematic diagram showing an example of an application scenario of the spatial positioning method according to the present disclosure.

[0027] like Figure 1 As shown, the application scenario of the spatial positioning method includes at least a second device and a first device. The second device and the first device can be communicatively connected.

[0028] The second device can determine its own spatial positioning in the global coordinate system, which can be a position or a posture. The second device can obtain spatial positioning through SLAM (simultaneous localization and mapping), motion capture system, outside-in tracking technology, etc. When the second device has a SLAM function, the second device may include at least one camera and at least one IMU. In one example, the second device may include a head-mounted device, which can be used to display a virtual object provided by the first device. Furthermore, the second device may include smart glasses, such as AR glasses, virtual reality (VR) glasses, etc.

[0029] It can be understood that the term "position" in this article can be expressed using a spatial rectangular coordinate system, while the term "pose" in this article describes the position and posture of an object, for example, using Euler angles, quaternions, etc. to represent the posture.

[0030] The first device may be a terminal device having a camera for capturing images, and the first device may include a handheld device, such as a mobile phone. The handheld device may be used to control a virtual object displayed by a head mounted device and includes a camera for capturing the first image.

[0031] The execution subject of the spatial positioning method provided by the present disclosure may be the second device, the first device, or the second device and the first device. For example, in an application scenario where a mobile phone is connected to AR glasses, the spatial positioning method may be executed by the mobile phone, which can save the computing power of the AR glasses and reduce the power consumption of the AR glasses. The following is an example in which the second device is AR glasses and the first device is a mobile phone.

[0032] Figure 2 A flowchart of an example 200 of a spatial positioning method according to the present disclosure is shown. Figure 2 The spatial positioning method shown can be executed by a first device.

[0033] like Figure 2 As shown, at 210, a second corresponding relationship between the second device and the 3D model of the second device may be acquired.

[0034] In the present disclosure, a 3D model may include 3D points and descriptors corresponding to the 3D points. A descriptor is description information of each feature point, and a descriptor may be used to distinguish each feature point, so that the corresponding feature point may be determined according to each descriptor. In the present disclosure, a descriptor may be a 3D descriptor or a 2D descriptor.

[0035] The 3D model of the second device may be composed of various 3D points, and the 3D model composed of all 3D points may be used to characterize the second device. Figure 3 A schematic diagram of an example of a 3D model of a second device according to the present disclosure is shown. For example, the second device is an AR glasses, and the 3D model formed is as follows: Figure 3 As shown, it is used to characterize the AR glasses.

[0036] The second device can be characterized based on the 3D model, and each 3D point on the 3D model corresponds to each feature point on the second device. For the one-to-one corresponding 3D point and feature point, the position of the 3D point on the 3D model is the same as the position of the feature point on the second device. For example, if the second device is an AR glasses, a feature point is located in the middle of the nose bridge of the AR glasses, and a 3D point is located in the middle of the nose bridge of the AR glasses represented by the 3D model, then the feature point on the AR glasses corresponds to the 3D point on the 3D model.

[0037] The one-to-one correspondence between the 3D points and the feature points can constitute a second correspondence relationship, and the second correspondence relationship can be used to characterize the mapping relationship between the second device and the 3D model. The determined second correspondence relationship may include a second correspondence relationship between all or part of the feature points on the AR glasses and the 3D points on the 3D model.

[0038] In the present disclosure, the 3D model of the second device may correspond to a 3D model coordinate system, and the 3D model coordinate system may be used to determine the relative position of the 3D model. The 3D model coordinate system may be constructed based on the 3D model, and the 3D model coordinate system may be a spatial rectangular coordinate system.

[0039] The 3D model coordinate system can be created with the second device as a reference, and a fixed position point on the second device (hereinafter referred to as the first position point) can be used as a position point in the 3D model coordinate system (hereinafter referred to as the second position point), and then the 3D model coordinate system is created based on the correspondence between the two position points. In one example, the coordinate origin in the 3D model coordinate system can be determined as the second position point.

[0040] In one example, a sensor in the second device may be determined as the first position point. The sensor may be a sensor used by the second device to perform SLAM, such as an IMU, a camera, etc. In another example, other position points on the second device that have a fixed relative position relationship with the sensor may be determined as the first position point.

[0041] In one example of the present disclosure, the 3D model of the second device may be pre-created, and the device that pre-creates the 3D model may be another device other than the second device.

[0042] In another example of the present disclosure, the 3D model of the second device can be created by the second device, and the second device can create the 3D model in real time. When the second device needs to create a 3D model, it can first determine a fixed position point on the second device as a position point in the 3D model coordinate system. For example, the camera on the second device can be used as the coordinate origin in the 3D model coordinate system. Then, based on the determined position point, a 3D model coordinate system is established, and then the sensor on the second device is used to create the 3D model of the second device in the 3D model coordinate system.

[0043] The created 3D model can be pre-stored in the second device, or in the first device, or in other devices, servers or clouds that can communicate with the second device and / or the first device. When the 3D model of the second device is needed, the 3D model can be obtained from the corresponding storage.

[0044] It can be understood that the step of acquiring the second corresponding relationship between the second device and the 3D model of the second device in 210 may not be a fixed step for executing the method of the present application.

[0045] In an example of obtaining the second corresponding relationship, the second corresponding relationship may be pre-created, and the created second corresponding relationship may be pre-stored in the second device, or in the first device, or in other devices, servers, or clouds that can communicate with the second device and / or the first device. In this example, the second corresponding relationship may be directly obtained from the corresponding storage.

[0046] In another example of obtaining the second correspondence, the second correspondence may be created in real time. In this example, when the second correspondence needs to be obtained, the feature points on the second device and the 3D points on the 3D model may be determined first, and then the correspondence between the feature points on the second device and the 3D points on the 3D model may be established, and the correspondence is the second correspondence.

[0047] At 220 , a first image of a second device may be acquired.

[0048] In the present disclosure, the first image of the second device captured by the first device may include the entire second device, or may include a portion of the second device. The first image may include 2D points of the second device and descriptors corresponding to the 2D points, and each image point of the first image may be represented by a 2D point and a descriptor corresponding to the 2D point. Based on this, the second device in the first image may be composed of multiple 2D points.

[0049] In the present disclosure, the first image of the second device and the 3D model of the second device are both used to represent the same object (i.e., the second device), and the 2D point on the second device in the first image can correspond to the 3D point on the 3D model. The corresponding 2D point and 3D point represent the same position point on the second device.

[0050] In the present disclosure, the first device may only shoot one image, namely the first image; or may shoot continuously, and the first image is one of the continuous multiple frame images. When the first device shoots continuously, the first image may be the first frame image, or may be a frame image generated after the first frame image, in which case there is a previous frame image before the first image is generated. For example, if the first image is the fifth frame image, the previous frame images include the first to fourth frame images, wherein the fourth frame image is the previous frame image of the first image.

[0051] In one example of the present disclosure, when the first image is not the first frame image, the previous frame image of the first image and the first image may be continuous frame images, and each previous frame image also includes the second device. The continuous frame images may continuously record the position change of the second device, and the position change of the second device changes continuously in the continuous frame images. Based on this, the first image may be searched according to the position of the second device in the previous frame image of the first image to determine the current position of the 2D point of the second device in the first image.

[0052] In this example, the previous frame image used may be a previous frame image of the first image, and may also include multiple continuous previous frame images, and the multiple previous frame images and the first image may be continuous frame images. The position of the second device in the previous frame image can be determined. When the previous frame image used includes multiple frames, each frame image in the previous frame image can determine the position of the corresponding second device. The following is described by taking a previous frame image as an example.

[0053] The estimated movement range of the first device can be determined according to the predetermined movement speed of the first device and the position of the second device in the previous frame image. The predetermined movement speed of the first device can be obtained by the IMU in the first device.

[0054] After determining the predetermined moving speed of the first device through the IMU of the first device, the time interval between the image frames can be determined. Multiplying the predetermined moving speed and the time interval can obtain the moving distance of the first device from the moment of the previous frame image to the moment of the first image. This moving distance can be equivalent to the distance between the position of the second device in the previous frame image and the position in the first image.

[0055] Therefore, the estimated moving range of the first device can be determined based on the moving distance of the first device and the position of the second device in the previous frame image. The range of the circle determined by taking the position of the second device in the previous frame image as the dot and the moving distance of the first device as the radius is the estimated moving range of the first device.

[0056] Then, a search is performed within the estimated moving range in the first image to determine the current position of the 2D point of the second device in the first image. For example, the position of the second device in the previous frame image and the relative position of the position in the first image can be determined. For example, in the previous frame image, the second device is located in the middle position, and the relative position of the second device in the previous frame image in the first image is also the middle position. Then, a search is performed within the estimated moving range centered on the determined relative position in the first image to determine the current position of the 2D point of the second device in the first image.

[0057] In this example, the estimated movement range in the first image is smaller than the search range of the entire first image, which narrows the search range and thus reduces the amount of data processing, thereby improving the efficiency of determining the 2D point of the second device from the first image.

[0058] In one example, the estimated movement range of the first device may be determined based on a predetermined movement speed and movement direction of the first device and a position of the second device in a previous frame image.

[0059] In this example, the moving direction of the first device can be obtained by the IMU on the first device. The moving direction of the first device is the moving direction in space, and the moving direction of the first device is opposite to the moving direction of the second device on the image captured by the first device. For example, if the first device moves upward in space, the second device moves downward on the image captured by the first device, for example, the position of the second device in the first image moves downward relative to the position in the previous frame image.

[0060] Based on the moving direction of the first device, the determined estimated moving range may be located in the direction opposite to the moving direction of the first device in the previous frame image, thereby further reducing the estimated moving range and further narrowing the search range.

[0061] In another example of the present disclosure, the relative position of the second device and the first device may be determined according to the position of the second device in the world coordinate system and the position of the first device in the world coordinate system, where the relative position is relative to the world coordinate system.

[0062] Then, the estimated range of the 2D point of the second device in the first image is calculated by the relative position of the second device and the first device. For example, the relative distance between the second device and the first device can be determined by the relative position of the second device and the first device, and then the shooting direction of the camera on the first device can be determined according to the posture of the first device and the position of the camera on the first device. Then, the shooting range of the camera can be determined according to the shooting direction of the camera and the field of view of the camera. The shooting range can be a conical area range with the camera as the vertex. Within the shooting range of the camera, at positions with different distances from the first device, the plane shooting range in the plane parallel to the lens plane of the camera to which the position belongs is different, and the farther the distance from the first device, the larger the plane shooting range. Thus, according to the relative distance between the second device and the first device and the shooting range of the camera, the plane shooting range in the plane where the second device is located can be determined. Then, according to the determined plane shooting range and the position of the second device in space, the position of the second device in the plane shooting range can be determined, and the position of the second device in the plane range and the range determined by the specified distance centered on the position can be determined as the estimated range of the 2D point of the second device in the first image.

[0063] After determining the estimated range of the 2D point of the second device in the first image, a search may be performed within the estimated range in the first image to determine the current position of the 2D point of the second device in the first image.

[0064] Through the above two examples, by estimating the range of the 2D point of the second device in the first image, it is only necessary to search within the estimated range in the first image, which narrows the scope of the local search and reduces the amount of data processing, thereby improving the efficiency of determining the 2D point of the second device from the first image.

[0065] Back to Figure 2 At 230, feature point matching is performed on the 2D points of the second device and the 3D points on the 3D model of the second device using the descriptors of the 3D points on the 3D model of the second device and the descriptors corresponding to the 2D points, so as to obtain a first correspondence between at least three non-collinear 2D points of the second device and the 3D points of the 3D model of the second device.

[0066] In the present disclosure, the 3D points on the 3D model corresponding to each 2D point on the second device in the first image can be determined by feature point matching, and a first correspondence relationship is formed between the 2D point and the corresponding 3D point. The correspondence relationship between at least three non-collinear 2D points and 3D points can be determined by feature point matching.

[0067] For example, the second device is an AR pair of glasses, and the three non-collinear points on the AR pair of glasses may be: points A and B at the bottom of the two frames, and point C in the middle of the bridge of the nose. The 2D points on the second device in the first image that are used to represent the points at the bottom of the two frames are A1 and B1, and the 2D points that are used to represent the points in the middle of the bridge of the nose are C1. The 3D points on the 3D model of the second device that are used to represent the points at the bottom of the two frames are A2 and B2, and the 3D points that are used to represent the points in the middle of the bridge of the nose are C2. Then A1 and A2 that are used to represent the points at the bottom of the same frame may constitute a first corresponding relationship, B1 and B2 that are used to represent the points at the bottom of the same frame may constitute another first corresponding relationship, and C1 and C2 that are used to represent the points in the middle of the bridge of the nose may constitute yet another first corresponding relationship.

[0068] Figure 4 A schematic diagram showing an example of a mapping relationship between 2D points in a first image and 3D points of a 3D model according to the present disclosure. Figure 4 As shown, F1, F2 and F3 are 3D points on the 3D model, f1, f2 and f3 are 2D points of the second device on the first image, and f1, f2 and f3 are not collinear. Among them, F1 and f1 represent the same position point on the second device, and F1 and f1 are mapped to each other in a first corresponding relationship. Correspondingly, F2 and f2 represent the same position point on the second device, and F2 and f2 are mapped to each other in a first corresponding relationship; F3 and f3 represent the same position point on the second device, and F3 and f3 are mapped to each other in a first corresponding relationship.

[0069] In one example, feature point matching can be performed based on the descriptor of the feature point. The descriptor of the 3D point of the 3D model that matches the descriptor can be determined from the descriptors of each 2D point on the second device in the first image, and the 2D points and 3D points corresponding to the two matching descriptors are matched with each other, and the matching 2D points and 3D points form a first corresponding relationship. Among them, two identical or similar descriptors can be considered to match each other. In another example, the method of feature point matching can include fast-orb, sift, etc.

[0070] It should be noted that the order of the operations of 210, 220 and 230 is not limited. Figure 2 The operations of 220 and 230 may be performed first, and then the operation of 210 may be performed; the operation of 210 may also be performed while the operations of 220 and 230 are performed.

[0071] At 240 , the position of the 3D model of the second device in the world coordinate system is determined according to the position of the second device in the world coordinate system and the second corresponding relationship between the second device and the 3D model.

[0072] In the present disclosure, the positioning of the second device in the world coordinate system may be a posture, and accordingly, the positioning of the determined 3D model in the world coordinate system is a posture. Figure 5 Provide detailed explanation.

[0073] The positioning of the second device in the world coordinate system may also be a position. Accordingly, the positioning of the determined 3D model in the world coordinate system is a position. Figure 7 Provide detailed explanation.

[0074] At 250 , the position and posture of the first device in the world coordinate system is determined according to the positioning of the 3D model of the second device in the world coordinate system and the first corresponding relationship.

[0075] The determined position of the first device may be a 6DOF (degree of freedom) position, including six degrees of freedom: front-to-back, up-down, left-to-right, pitch, yaw, and roll. The spatial positioning of the first device interacting with the second device is represented by a 6DOF position, thereby improving the accuracy of the spatial positioning of the first device.

[0076] Figure 5 A flowchart of another example 500 of a spatial positioning method according to the present disclosure is shown. Figure 5 The spatial positioning method shown can be executed by a first device.

[0077] like Figure 5 As shown, at 510, a first image of the second device may be acquired, where the first image includes 2D points of the second device and descriptors corresponding to the 2D points.

[0078] At 520, feature point matching is performed on the 2D points of the second device and the 3D points on the 3D model of the second device using descriptors corresponding to the 3D points and descriptors corresponding to the 2D points, so as to obtain a first correspondence between at least three non-collinear 2D points of the second device and the 3D points of the 3D model of the second device.

[0079] Figure 5 The operations of 510 and 520 are respectively the same as those described above. Figure 2 The operations of 220 and 230 are similar and will not be described here.

[0080] At 530 , the position and pose of the 3D model of the second device in the world coordinate system is determined according to the position and pose of the second device in the world coordinate system and the second corresponding relationship.

[0081] In the present disclosure, the second device may have SLAM computing capability, so that the second device may calculate the pose of the second device by SLAM. In one example, the second device may calculate the real-time pose of the second device by SLAM in real time, and when the pose information of the second device is needed, it may be directly obtained from the second device. In another example, the second device may be triggered to perform SLAM calculation by triggering, and the second device does not perform SLAM calculation when not triggered.

[0082] The second correspondence is the correspondence between the feature point on the second device and the 3D point on the 3D model of the second device, that is, the second correspondence is used to represent the correspondence between the second device and the 3D model. After obtaining the position and posture of the second device in the world coordinate system, based on the second correspondence and the position and posture of the second device in the world coordinate system, the relationship between the second device, the 3D model and the world coordinate system can be determined as follows: Figure 6 As shown, Figure 6 FIG. 2 shows a schematic diagram of an example relationship between a second device, a 3D model, and a world coordinate system according to the present disclosure. Figure 6 As shown, L1 represents the position and posture of the second device in the world coordinate system, and L2 represents the position and posture of the 3D model in the world coordinate system. L2 can be determined by L1 and the known second corresponding relationship.

[0083] It should be noted that the operation of 530 can be performed as a step in the spatial positioning method, that is, the operation of 530 is performed each time the spatial positioning method is executed. In another example, the operation of 530 can be performed by other devices. For example, when the spatial positioning method of the present invention is performed by a first device, the operation of 530 can be performed by a second device, and when the first device needs the pose information of the 3D model in the world coordinate system, it can be obtained from the second device. When the spatial positioning method of the present invention is performed by a second device, the operation of 530 can be performed by the first device, and when the second device needs the pose information of the 3D model in the world coordinate system, it can be obtained from the first device. In addition, the operation of 530 can also be performed by other devices other than the second device and the first device.

[0084] At 540 , a PnP (perspective-n-point) algorithm is used to calculate the position and posture of the first device in the 3D model coordinate system according to the first corresponding relationship.

[0085] In the present disclosure, the first correspondence relationship is a correspondence relationship between a 2D point in the first image and a 3D point of the 3D model.

[0086] In the present disclosure, the PnP algorithm may include a P3P algorithm, an EPnP (Efficient PnP) algorithm, an aP3P (Algebraic Solution to the Perspective-Three-Point) algorithm, etc. In one example, the PnP algorithm used in the present disclosure may be a PnP algorithm based on the least squares method.

[0087] Based on the known first correspondence, at least three pairs of 2D points and 3D points that conform to the first correspondence can be obtained. Through the PnP algorithm, the position and posture of the first device in the 3D model coordinate system can be calculated according to the coordinates of the 2D points in the camera coordinate system and the coordinates of the 3D points in the 3D model coordinate system.

[0088] In one example, a RANSAC (random sample consensus) algorithm may be used to determine the inliers from a first correspondence between at least three non-collinear 2D points of the second device in the first image and 3D points of the 3D model of the second device.

[0089] In all points of the first correspondence, multiple inliers are randomly assumed as initial values, and the multiple inliers are fitted into a model. The model is adapted to the multiple inliers as initial values, and the parameters of the model are calculated from the multiple inliers. Next, the model is used to test other feature points in the first correspondence. If one of the other feature points is suitable for the model, it can be determined that the feature point is an inlier. Otherwise, it can be considered that the feature point is not an inlier. In this way, the number of inliers can be expanded. After testing all feature points, the model is re-evaluated using all feature points determined as inliers to update the model. The evaluation method can be to evaluate using the error rate of the model. The above process is an iterative process. After multiple iterations, in each iteration, if there are too few inliers and they are not as good as the model in the previous iteration, the model of the iteration can be discarded; if the model generated by the iteration is worse than the model of the previous iteration, the model of the iteration can be retained and the next iteration can be performed.

[0090] After the inner points are determined, a PnP algorithm may be used to calculate the pose of the first device in the 3D model coordinate system of the second device based on the inner points.

[0091] Through the RANSAC algorithm in this example, feature points with higher matching degrees can be screened out from all feature points matched by the feature points in the first correspondence as inliers. After the RANSAC algorithm, the screened inliers not only have higher feature point matching degrees, but also have fewer feature points used for the PnP algorithm, thereby reducing the amount of data calculation on the basis of improving the accuracy of pose calculation.

[0092] Back to Figure 5 At 550, based on the pose of the first device in the 3D model coordinate system of the second device and the pose of the 3D model of the second device in the world coordinate system, the pose of the first device in the world coordinate system is obtained.

[0093] In the present disclosure, a 3D model coordinate system is constructed based on a 3D model, and the relative position between the 3D model coordinate system and the 3D model is fixed. According to the position of the first device in the 3D model coordinate system and the relative position relationship between the 3D model coordinate system and the 3D model, the position of the first device relative to the 3D model can be determined. Then, based on the position of the first device relative to the 3D model and the position of the 3D model in the world coordinate system, the position of the first device in the world coordinate system can be obtained.

[0094] Figure 7 A flowchart of another example 700 of a spatial positioning method according to the present disclosure is shown. Figure 7 The spatial positioning method shown can be executed by a first device.

[0095] like Figure 7 As shown, at 710, a first image of the second device may be acquired, where the first image includes 2D points of the second device and descriptors corresponding to the 2D points.

[0096] At 720, feature point matching is performed on the 2D points of the second device and the 3D points on the 3D model of the second device using descriptors corresponding to the 3D points and descriptors corresponding to the 2D points to obtain a first correspondence between at least three non-collinear 2D points of the second device and the 3D points of the 3D model of the second device.

[0097] Figure 7 The operations of 710 and 720 are respectively the same as those described above. Figure 2 The operations of 220 and 230 are similar and will not be described here.

[0098] At 730 , the position of the 3D point of the 3D model of the second device in the world coordinate system is determined according to the position of the second device in the world coordinate system and the second corresponding relationship.

[0099] In this example, the position of the second device in the world coordinate system may include three degrees of freedom: front and back, up and down, and left and right. For example, in a rectangular coordinate system, the position of the second device may be represented by values ​​on the X-axis, the Y-axis, and the Z-axis.

[0100] In this example, the second device may have a positioning capability, such as GPS, Beidou, etc. The second device may obtain its own position information in the world coordinate system through positioning.

[0101] After determining the position of the 3D model in the world coordinate system, at 740, the pose of the first device in the world coordinate system is calculated using a PnP algorithm according to the first correspondence and the position of the 3D point of the 3D model of the second device that conforms to the first correspondence in the world coordinate system.

[0102] In this example, the 3D points targeted by the PnP algorithm are all referenced to the world coordinate system, so the obtained position and posture of the first device is also referenced to the world coordinate system. In one example, the PnP algorithm used may be a PnP algorithm based on the least squares method.

[0103] In one example, the positions of all or part of the 3D points in the first corresponding relationship in the world coordinate system may be determined, and then the PnP algorithm calculation is performed on the 3D points with the world coordinate system as a reference to obtain the position and posture of the first device in the world coordinate system.

[0104] In one example, before performing the operation of 740, an inlier point may be determined from a first correspondence between at least three non-collinear 2D points of the second device in the first image and a 3D point of the 3D model of the second device using a RANSAC algorithm. Then, the determined inlier point is determined as a point to be used when the first correspondence is applied to the PnP algorithm, that is, the pose of the first device in the world coordinate system is calculated using the PnP algorithm based on the determined first correspondence between the inlier point and the position of the inlier point in the 3D point of the 3D model of the second device in the world coordinate system.

[0105] In one example of the present disclosure, when the first device cannot capture a first image including the second device, it can be determined that the first image captured by the first device does not include at least three 2D points of the second device that are not collinear. In addition, the first device can capture a first image including the second device, and there are less than three 2D points of the second device in the captured first image, or no less than three 2D points of the second device in the first image are collinear, then it can be determined that the first image captured by the first device does not include at least three 2D points of the second device that are not collinear.

[0106] In the case where the first image captured by the first device does not include at least three non-collinear 2D points of the second device, the first device may capture a second image including the designated object, wherein the second image may include the 2D points of the designated object that may serve as key points, and the relative position of the designated object and the second device is fixed.

[0107] In one example, the second device may include a head-mounted device, and in this case, the designated object may include a human face. When the user wears the head-mounted device, the head-mounted device and the user's face have a fixed relative position. In another example, the designated object may also include designated organs on the human face, such as eyes, nose, ears, mouth, etc.

[0108] In one example, the human faces included in the designated object may include a universal human face, and the universal human face is used to establish a 3D human face model of the user using the head mounted device. No matter who wears the head mounted device, the universal human face 3D model is used, which can save the operation of modeling the user's face. In another example, the human faces included in the designated object may include the face of a designated user, and the designated user may be a user using the second device, that is, a dedicated human face 3D model needs to be established for each user wearing the head mounted device.

[0109] In this example, a 3D model of a specified object may be pre-built and stored. In one example, the stored 3D model may include 3D points and corresponding descriptors.

[0110] Since the relative position between the designated object and the second device is fixed, the relative position relationship between the pre-stored 3D model of the designated object and the 3D model of the second device can be established based on the relative position. For example, the second device is a head-mounted device, and the designated object is a human face. When the user wears the head-mounted device, the relative position difference between the human face and the head-mounted device is L, then the position difference between the 3D model of the human face and the 3D model of the head-mounted device can be determined to be L1. After calculating the position or posture of the 3D model of the head-mounted device in the world coordinate system, the position or posture of the 3D model of the human face in the world coordinate system can be calculated by L1.

[0111] Key point detection is performed on at least three non-collinear 2D points of the specified object in the second image to obtain a matching relationship between the at least three non-collinear 2D points of the specified object and the 3D points of the 3D model of the specified object to obtain a third corresponding relationship between the 3D model of the specified object and the second image.

[0112] In this example, the key points of the specified object may be specified. For example, when the specified object is a face, the key points specified on the face may include feature points at the corners of the mouth, the tip of the nose, and the like.

[0113] After detecting 2D points corresponding to key points on a specified object by means of key point detection, the 2D points corresponding to the key points may be matched with 3D points of a 3D model of the specified object to establish a matching relationship.

[0114] In one example, the descriptors of the 2D points corresponding to each key point in the second image can be matched with the descriptors of each 3D point on the 3D model of the specified object. The 2D points and 3D points whose descriptors match represent the same key point on the specified object, and the 2D points and the 3D points form a matching relationship.

[0115] The matching relationships corresponding to the key points are combined to obtain a third corresponding relationship between the 3D model of the specified object and the second image.

[0116] Through the above examples, when the second device cannot be captured or the first image captured by the first device does not include at least three non-collinear 2D points of the second device, the position of the first device in the world coordinate system can be calculated by capturing a specified object whose relative position to the second device is fixed, thereby improving the robustness of spatial positioning.

[0117] In another example of the present disclosure, when a first image captured by a first device includes a designated object and a second device, the first image may include a 2D point of the designated object that can be used as a key point. The relative position of the designated object and the second device is fixed.

[0118] In this example, key point detection is performed on at least three non-collinear 2D points of the specified object in the second image to obtain a matching relationship between the at least three non-collinear 2D points of the specified object and the 3D points of the 3D model of the specified object. The 3D model of the specified object includes 3D points and key points corresponding to the 3D points. The obtained matching relationship is a matching relationship for the key points of the specified object.

[0119] After obtaining the matching relationship corresponding to the key points, the PnP algorithm can be used to calculate the pose of the first device in the 3D model coordinate system of the second device according to the first corresponding relationship and the matching relationship, and then based on the pose of the first device in the 3D model coordinate system of the second device and the pose of the 3D model of the second device in the world coordinate system, the pose of the first device in the world coordinate system is obtained.

[0120] In one example, before using the PnP algorithm, the coordinate system of the 3D model of the specified object can be converted to the coordinate system of the 3D model of the second device based on the relative position relationship between the pre-stored 3D model of the specified object and the 3D model of the second device, so that the 3D model of the specified object and the 3D model of the second device are unified in the coordinate system of the 3D model of the second device, which is convenient for performing the PnP algorithm.

[0121] Through the above example, a more accurate posture can be obtained by combining the posture calculated by the second device with the posture calculated by the specified object.

[0122] In one example of the present disclosure, the first device may be configured with an IMU, and IMU data may be collected in real time through its own IMU. The first device may use the collected IMU data to calculate the pose (hereinafter referred to as pose a) of the first device in the world coordinate system in real time.

[0123] The posture of the first device is different at different times, so the posture a calculated based on the IMU data at different times may be different.

[0124] When the first device captures the first image, the current posture a can be calculated based on the currently collected IMU data. Then, the posture of the first device in the world coordinate system (hereinafter referred to as posture b) is determined based on the first image through the embodiment of the present disclosure, and the posture a and the posture b are fused to obtain the posture of the first device. The fusion method may include averaging, etc.

[0125] In an example, the collected IMU data may be directly fused with the position and posture of the first device in the world coordinate system determined according to the first image.

[0126] In this example, the accuracy of the pose of the first device is improved by fusing the poses calculated by two different methods. In addition, the IMU data is continuous, and the pose determined by the first image is discontinuous. On the basis of the pose obtained by the method provided in the present disclosure, the pose calculated by the IMU data is fused to obtain a higher frequency pose and reduce data latency.

[0127] Figure 8 FIG. 8 is a block diagram showing an example of a space positioning device 800 according to the present disclosure. The space positioning device 800 may be applied to a first device.

[0128] like Figure 8 As shown, the spatial positioning device 800 includes an image acquisition unit 810 , a feature point matching unit 820 , a positioning unit 830 and a posture determination unit 840 .

[0129] The image acquisition unit 810 is configured to acquire a first image of the second device, where the first image includes 2D points of the second device and descriptors corresponding to the 2D points.

[0130] The feature point matching unit 820 is configured to perform feature point matching on the 2D points of the second device and the 3D points on the 3D model of the second device using the descriptors corresponding to the 3D points and the descriptors corresponding to the 2D points on the 3D model of the second device to obtain a first correspondence between at least three non-collinear 2D points of the second device and the 3D points of the 3D model of the second device, wherein the 3D model of the second device includes the 3D points and the descriptors corresponding to the 3D points.

[0131] The positioning unit 830 is configured to determine the positioning of the 3D model of the second device in the world coordinate system according to the positioning of the second device in the world coordinate system and the second corresponding relationship between the second device and the 3D model.

[0132] The posture determination unit 840 is configured to determine the posture of the first device in the world coordinate system according to the positioning of the 3D model of the second device in the world coordinate system and the first corresponding relationship.

[0133] In one example, the spatial positioning device 800 also includes a key point matching unit. The image acquisition unit is configured to acquire a second image including a designated object when the first image acquired by the first device does not include at least three non-collinear 2D points of the second device, the second image includes 2D points of the designated object that can be used as key points, and the relative position of the designated object and the second device is fixed. The positioning unit is configured to determine the positioning of the 3D model of the designated object in the world coordinate system based on the positioning of the 3D model of the second device in the world coordinate system and the relative position relationship between the pre-stored 3D model of the designated object and the 3D model of the second device, wherein the 3D model of the designated object includes 3D points and key points corresponding to the 3D points. The key point matching unit is configured to perform key point detection on at least three non-collinear 2D points of the designated object in the second image, obtain a matching relationship between at least three non-collinear 2D points of the designated object and the 3D points of the 3D model of the designated object, so as to obtain a third corresponding relationship between the second image and the 3D model of the designated object. The posture determination unit is configured to determine the posture of the first device in the world coordinate system based on the positioning of the 3D model of the specified object in the world coordinate system and the third corresponding relationship.

[0134] In one example, the spatial positioning device 800 also includes a key point matching unit, which is configured to perform key point detection on at least three non-collinear 2D points of the specified object in the second image to obtain a matching relationship between the at least three non-collinear 2D points of the specified object and the 3D points of the 3D model of the specified object, wherein the 3D model of the specified object includes 3D points and key points corresponding to the 3D points. The posture determination unit is configured to: use a PnP algorithm to calculate the posture of the first device in the 3D model coordinate system of the second device according to the first corresponding relationship and the matching relationship; and obtain the posture of the first device in the world coordinate system based on the posture of the first device in the 3D model coordinate system of the second device and the posture of the 3D model of the second device in the world coordinate system.

[0135] In one example, the spatial positioning device 800 also includes a coordinate system conversion unit, which is configured to convert the coordinate system of the 3D model of the specified object into the coordinate system of the 3D model of the second device based on the relative position relationship between the pre-stored 3D model of the specified object and the 3D model of the second device.

[0136] In one example, the image acquisition unit is configured to: determine the position of the second device in a previous frame image; determine an estimated moving range of the first device based on a predetermined moving speed of the first device and the position of the second device in a previous frame image; and search within the estimated moving range in the first image to determine the current position of the 2D point of the second device in the first image.

[0137] In one example, the image acquisition unit is configured to: determine the relative position of the second device and the first device based on the position of the second device in the world coordinate system and the position of the first device in the world coordinate system; calculate the estimated range of the 2D point of the second device in the first image through the relative position of the second device and the first device; and search within the estimated range in the first image to determine the current position of the 2D point of the second device in the first image.

[0138] Fig. 9 A block diagram of another example of a spatial positioning device 800 according to the present disclosure is shown.

[0139] like Fig. 9 As shown, the positioning unit 830 may include a first pose determination module 831, which is configured to determine the pose of the 3D model of the second device in the world coordinate system according to the pose of the second device in the world coordinate system and the second corresponding relationship.

[0140] The posture determination unit 840 may include a second posture determination module 841 and a third posture determination module 842. The second posture determination module 841 may be configured to calculate the posture of the first device in the 3D model coordinate system of the second device according to the first corresponding relationship using the PnP algorithm. The third posture determination module 842 may be configured to obtain the posture of the first device in the world coordinate system based on the posture of the first device in the 3D model coordinate system of the second device and the posture of the 3D model of the second device in the world coordinate system.

[0141] In this example, the posture determination module 831, the second posture determination module 841 and the third posture determination module 842 may be the same module or different modules.

[0142] In one example, the spatial positioning device 800 may also include an inlier determination unit, which is configured to determine an inlier from a first correspondence between at least three non-collinear 2D points of the second device in the first image and a 3D point of the 3D model of the second device using a random sampling consistency algorithm; and determine the inlier as a point to be used when the first correspondence is applied to the PnP algorithm.

[0143] Fig.10A block diagram of another example of a spatial positioning device 800 according to the present disclosure is shown.

[0144] like Fig.10 As shown, the positioning unit 830 may include a position determination module 832, which may be configured to determine the position of the 3D point of the 3D model of the second device in the world coordinate system according to the position of the second device in the world coordinate system and the second corresponding relationship.

[0145] The posture determination unit 840 may include a fourth posture determination module 843, which may be configured to calculate the posture of the first device in the world coordinate system using a PnP algorithm based on the first correspondence and the position of the 3D point of the 3D model of the second device that conforms to the first correspondence in the world coordinate system.

[0146] Reference above Figures 1 to 10 , embodiments of the method and apparatus for spatial positioning according to the present disclosure are described.

[0147] The device for spatial positioning disclosed in the present invention can be implemented by hardware, software, or a combination of hardware and software. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the device in which it is located reading the corresponding computer program instructions in the memory into the memory and running it. In the present disclosure, the device for spatial positioning can be implemented, for example, using an electronic device.

[0148] Fig.11 A block diagram of an electronic device 1100 for implementing a spatial positioning method according to an embodiment of the present disclosure is shown.

[0149] like Fig.11 As shown, the electronic device 1100 may include at least one processor 1110, a memory (e.g., a non-volatile memory) 1120, a memory 1130, and a communication interface 1140, and the at least one processor 1110, the memory 1120, the memory 1130, and the communication interface 1140 are connected together via a bus 1150. At least one processor 1110 executes at least one computer-readable instruction stored or encoded in the memory (i.e., the above-mentioned element implemented in the form of software).

[0150] In one embodiment, computer executable instructions are stored in a memory, which when executed cause at least one processor 1110 to: capture a first image of a second device, the first image including 2D points of the second device and descriptors corresponding to the 2D points; perform feature point matching on the 2D points of the second device and the 3D points on the 3D model of the second device using descriptors corresponding to the 3D points on the 3D model of the second device to obtain a first correspondence between at least three non-collinear 2D points of the second device and 3D points of the 3D model of the second device, the 3D model of the second device including 3D points and descriptors corresponding to the 3D points; determine the positioning of the 3D model of the second device in the world coordinate system based on the positioning of the second device in the world coordinate system and the second correspondence between the second device and the 3D model; and determine the posture of the first device in the world coordinate system based on the positioning of the 3D model of the second device in the world coordinate system and the first correspondence.

[0151] It should be understood that the computer executable instructions stored in the memory, when executed, cause at least one processor 1110 to perform the above combined operations in various embodiments of the present disclosure. Figure 1-10 Describes the various operations and functions.

[0152] According to one embodiment, a program product such as a machine-readable medium is provided. The machine-readable medium may have instructions (i.e., the above-mentioned elements implemented in the form of software), which, when executed by a machine, causes the machine to perform the above-mentioned combination of various embodiments of the present disclosure. Figure 1-10 Describes the various operations and functions.

[0153] A system or device equipped with a readable storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

[0154] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above-mentioned embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute a part of the present invention.

[0155] The computer program code required for the operation of each part of the present disclosure can be written in any one or more programming languages, including object-oriented programming languages, such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB, NET and Python, conventional procedural programming languages ​​such as C language, Visual Basic 2003, Perl, COBOL 2002, PHP and ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages. The program code can be run on the user's computer, or run on the user's computer as an independent software package, or run partly on the user's computer and another part on the remote computer, or all on the remote computer or server. In the latter case, the remote computer can be connected to the user's computer by any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service, such as software as a service (SaaS).

[0156] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0157] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0158] Not all steps and units in the above-mentioned processes and system structure diagrams are necessary, and some steps or units can be ignored according to actual needs. The execution order of each step is not fixed and can be determined according to needs. The device structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0159] The term "exemplary" as used throughout this disclosure means "serving as an example, instance, or illustration" and does not imply "preferred" or "advantageous" over other embodiments. The detailed description includes specific details for the purpose of providing an understanding of the described techniques. However, the techniques may be practiced without these specific details. In some instances, in order to avoid obscuring the concepts of the described embodiments, well-known structures and devices are shown in block diagram form.

[0160] The optional implementation modes of the embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the embodiments of the present disclosure are not limited to the specific details in the above implementation modes; within the technical concept of the embodiments of the present disclosure, a variety of simple modifications can be made to the technical solutions of the embodiments of the present disclosure, and these simple modifications all fall within the protection scope of the embodiments of the present disclosure.

[0161] The above description of the present disclosure is provided to enable any person of ordinary skill in the art to implement or use the present disclosure. Various modifications to the present disclosure will be apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other variations without departing from the scope of protection of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A method for spatial positioning, the method being performed by a first device, the method include: Acquire a first image of a second device, where the first image includes 2D points of the second device and descriptors corresponding to the 2D points; Performing feature point matching on the 2D points of the second device and the 3D points on the 3D model of the second device using descriptors corresponding to the 3D points on the 3D model of the second device and descriptors corresponding to the 2D points, so as to obtain a first correspondence between at least three non-collinear 2D points of the second device and 3D points of the 3D model of the second device, wherein the 3D model of the second device includes 3D points and descriptors corresponding to the 3D points, the 3D model of the second device is composed of 3D points, and the 3D model composed of all 3D points is used to characterize the second device; Determine the positioning of the 3D model of the second device in the world coordinate system according to the positioning of the second device in the world coordinate system and a second corresponding relationship between the second device and the 3D model, wherein the second corresponding relationship is used to characterize a mapping relationship between the second device and the 3D model; as well as The position and posture of the first device in the world coordinate system is determined according to the positioning of the 3D model of the second device in the world coordinate system and the first corresponding relationship.

2. The method according to claim 1, in, The positioning of the second device in the world coordinate system includes a posture, Determining the positioning of the 3D model of the second device in the world coordinate system according to the positioning of the second device in the world coordinate system and the second corresponding relationship includes: Determining the position and posture of the 3D model of the second device in the world coordinate system according to the position and posture of the second device in the world coordinate system and the second corresponding relationship; and Determining the position and posture of the first device in the world coordinate system according to the positioning of the 3D model of the second device in the world coordinate system and the first corresponding relationship includes: Calculate the pose of the first device in the 3D model coordinate system of the second device according to the first corresponding relationship using a PnP algorithm; and The pose of the first device in the world coordinate system is obtained based on the pose of the first device in the 3D model coordinate system of the second device and the pose of the 3D model of the second device in the world coordinate system.

3. The method according to claim 1, in, The positioning of the second device in the world coordinate system includes a position, Determining the positioning of the 3D model of the second device in the world coordinate system according to the positioning of the second device in the world coordinate system and the second corresponding relationship includes: Determining the position of the 3D point of the 3D model of the second device in the world coordinate system according to the position of the second device in the world coordinate system and the second corresponding relationship; and Determining the position and posture of the first device in the world coordinate system according to the positioning of the 3D model of the second device in the world coordinate system and the first corresponding relationship includes: The pose of the first device in the world coordinate system is calculated using a PnP algorithm according to the first correspondence and the position of the 3D point of the 3D model of the second device that conforms to the first correspondence in the world coordinate system.

4. The method according to claim 2 or 3, in, Before using the PnP algorithm, the method further includes: determining inliers from first correspondences between at least three non-collinear 2D points of the second device in the first image and 3D points of the 3D model of the second device using a random sampling consensus algorithm; and The interior point is determined as a point to be used when the first correspondence is applied to a PnP algorithm.

5. The method according to claim 2 or 3, in, The PnP algorithm used is a PnP algorithm based on the least squares method.

6. The method according to any one of claims 1 to 3, in, The second device includes a head-mounted device, which is used to display the virtual object provided by the first device. The first device includes a handheld device, which is used to control the virtual object displayed by the head-mounted device and includes a camera device for capturing a first image.

7. The method of claim 1, further comprising: include: When the first image captured by the first device does not include at least three non-collinear 2D points of the second device, capturing a second image including a designated object, the second image including the 2D points of the designated object that can be used as key points, and the relative position between the designated object and the second device is fixed; Determine the location of the 3D model of the specified object in the world coordinate system based on the location of the 3D model of the second device in the world coordinate system and the relative positional relationship between the pre-stored 3D model of the specified object and the 3D model of the second device, wherein the 3D model of the specified object includes 3D points and key points corresponding to the 3D points; Performing key point detection on at least three non-collinear 2D points of the designated object in the second image to obtain a matching relationship between the at least three non-collinear 2D points of the designated object and a 3D point of the 3D model of the designated object, so as to obtain a third corresponding relationship between the second image and the 3D model of the designated object; Based on the positioning of the 3D model of the designated object in the world coordinate system and the third corresponding relationship, the position and posture of the first device in the world coordinate system is determined.

8. The method according to claim 1, in, The first image further includes a designated object, the first image further includes a 2D point of the designated object that can be used as a key point, the relative position of the designated object and the second device is fixed, and the method further includes: Performing key point detection on at least three non-collinear 2D points of the specified object in the second image to obtain a matching relationship between the at least three non-collinear 2D points of the specified object and 3D points of a 3D model of the specified object, wherein the 3D model of the specified object includes the 3D points and key points corresponding to the 3D points; and Determining the position and posture of the first device in the world coordinate system according to the positioning of the 3D model of the second device in the world coordinate system and the first corresponding relationship includes: Calculate the pose of the first device in the 3D model coordinate system of the second device according to the first corresponding relationship and the matching relationship using a PnP algorithm; and The pose of the first device in the world coordinate system is obtained based on the pose of the first device in the 3D model coordinate system of the second device and the pose of the 3D model of the second device in the world coordinate system.

9. The method according to claim 8, in, Before using the PnP algorithm to calculate the pose of the first device in the 3D model coordinate system of the second device according to the first corresponding relationship and the matching relationship, the method further includes: According to the relative positional relationship between the pre-stored 3D model of the designated object and the 3D model of the second device, the coordinate system of the 3D model of the designated object is converted into the coordinate system of the 3D model of the second device.

10. The method according to claim 7 or 8, in, When the second device includes a head-mounted device, the designated object includes a human face.

11. The method according to any one of claims 1 to 3, in, Acquiring the first image of the second device includes: Determine the position of the second device in the previous frame image; determining an estimated movement range of the first device according to a predetermined movement speed of the first device and a position of the second device in the previous frame image; and A search is performed within the estimated movement range in the first image to determine a current position of the 2D point of the second device in the first image.

12. The method according to any one of claims 1 to 3, in, Acquiring the first image of the second device includes: Determining a relative position of the second device and the first device according to a position of the second device in a world coordinate system and a position of the first device in the world coordinate system; Calculating an estimated range of a 2D point of the second device in the first image according to a relative position of the second device and the first device; and A search is performed within the estimated range in the first image to determine a current position of the 2D point of the second device in the first image.

13. A device for spatial positioning, the device being applied to a first device, the device include: An image acquisition unit is configured to acquire a first image of a second device, wherein the first image includes 2D points of the second device and descriptors corresponding to the 2D points; a feature point matching unit, configured to perform feature point matching on the 2D points of the second device and the 3D points on the 3D model of the second device using descriptors corresponding to the 3D points on the 3D model of the second device and descriptors corresponding to the 2D points, so as to obtain a first correspondence between at least three non-collinear 2D points of the second device and 3D points of the 3D model of the second device, wherein the 3D model of the second device includes 3D points and descriptors corresponding to the 3D points, the 3D model of the second device is composed of 3D points, and the 3D model composed of all 3D points is used to characterize the second device; a positioning unit, configured to determine a positioning of the 3D model of the second device in the world coordinate system according to the positioning of the second device in the world coordinate system and a second corresponding relationship between the second device and the 3D model, wherein the second corresponding relationship is used to characterize a mapping relationship between the second device and the 3D model; as well as The posture determination unit is configured to determine the posture of the first device in the world coordinate system according to the positioning of the 3D model of the second device in the world coordinate system and the first corresponding relationship.

14. The device according to claim 13, in, The positioning of the second device in the world coordinate system includes a posture, The positioning unit comprises: a first pose determination module, configured to determine the pose of the 3D model of the second device in the world coordinate system according to the pose of the second device in the world coordinate system and the second corresponding relationship; and The posture determination unit comprises: A second posture determination module is configured to calculate the posture of the first device in the 3D model coordinate system of the second device according to the first corresponding relationship using a PnP algorithm; and The third posture determination module is configured to obtain the posture of the first device in the world coordinate system based on the posture of the first device in the 3D model coordinate system of the second device and the posture of the 3D model of the second device in the world coordinate system.

15. The device according to claim 13, in, The positioning of the second device in the world coordinate system includes a position, The positioning unit comprises: a position determination module, configured to determine the position of the 3D point of the 3D model of the second device in the world coordinate system according to the position of the second device in the world coordinate system and the second corresponding relationship; and The posture determination unit comprises: The fourth posture determination module is configured to calculate the posture of the first device in the world coordinate system using a PnP algorithm based on the first correspondence and the position of the 3D point of the 3D model of the second device that conforms to the first correspondence in the world coordinate system.

16. The device according to claim 14 or 15, further comprising: include: An inlier determination unit is configured to determine an inlier from a first correspondence between at least three non-collinear 2D points of the second device in the first image and a 3D point of the 3D model of the second device using a random sampling consistency algorithm; and determine the inlier as a point to be used when the first correspondence is applied to a PnP algorithm.

17. The device according to claim 13, in, The image acquisition unit is configured to acquire a second image including a specified object when the first image acquired by the first device does not include at least three non-collinear 2D points of the second device, the second image including the 2D points of the specified object that can be used as key points, and the relative position between the specified object and the second device is fixed; The positioning unit is configured to determine the positioning of the 3D model of the specified object in the world coordinate system based on the positioning of the 3D model of the second device in the world coordinate system and the relative positional relationship between the pre-stored 3D model of the specified object and the 3D model of the second device, wherein the 3D model of the specified object includes 3D points and key points corresponding to the 3D points; The apparatus further includes: a key point matching unit configured to perform key point detection on at least three non-collinear 2D points of the designated object in the second image, obtain a matching relationship between the at least three non-collinear 2D points of the designated object and a 3D point of the 3D model of the designated object, so as to obtain a third corresponding relationship between the second image and the 3D model of the designated object; The posture determination unit is configured to determine the posture of the first device in the world coordinate system based on the positioning of the 3D model of the designated object in the world coordinate system and the third corresponding relationship.

18. The device according to claim 13, in, The first image also includes a designated object, the first image also includes a 2D point of the designated object that can be used as a key point, the relative position of the designated object and the second device is fixed, and the apparatus further includes: a key point matching unit, configured to perform key point detection on at least three non-collinear 2D points of the specified object in the second image, and obtain a matching relationship between the at least three non-collinear 2D points of the specified object and 3D points of a 3D model of the specified object, wherein the 3D model of the specified object includes the 3D points and key points corresponding to the 3D points; and The posture determination unit is configured as follows: Calculate the pose of the first device in the 3D model coordinate system of the second device according to the first corresponding relationship and the matching relationship using a PnP algorithm; and The pose of the first device in the world coordinate system is obtained based on the pose of the first device in the 3D model coordinate system of the second device and the pose of the 3D model of the second device in the world coordinate system.

19. The device according to claim 18, further comprising: include: The coordinate system conversion unit is configured to convert the coordinate system of the 3D model of the specified object into the coordinate system of the 3D model of the second device according to the relative position relationship between the pre-stored 3D model of the specified object and the 3D model of the second device.

20. The device according to any one of claims 13 to 15, in, The image acquisition unit is configured as follows: Determine the position of the second device in the previous frame image; determining an estimated moving range of the first device according to a predetermined moving speed of the first device and a position of the second device in the previous frame image; and A search is performed within the estimated movement range in the first image to determine a current position of the 2D point of the second device in the first image.

21. The device according to any one of claims 13 to 15, in, The image acquisition unit is configured as follows: Determining a relative position of the second device and the first device according to a position of the second device in a world coordinate system and a position of the first device in the world coordinate system; Calculating an estimated range of a 2D point of the second device in the first image according to a relative position of the second device and the first device; and A search is performed within the estimated range in the first image to determine a current position of the 2D point of the second device in the first image.

22. An electronic device, include: At least one processor, and a memory coupled to the at least one processor, the memory storing instructions, which, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 12.

23. A non-volatile computer-readable storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 12 when executed by a processor.

Images