Multi-device positioning method, system and related device

Abstract

The invention discloses a multi-device positioning method, system and related device, and in the method, images collected by multiple devices for the same scene can be obtained, and the position relationship among the multiple devices is determined by using the images collected by the multiple devices, so that multi-device positioning is realized. It can be seen that according to the method, positioning between the devices is carried out through the images collected by the devices, the positioning result is not affected by the environment, positioning between the devices can be completed only through two devices, the application range of the method is wider, and the position relation between the devices can be rapidly and accurately obtained.

Description

Technical Field

[0001] The present application relates to the field of terminal and communication technology, and in particular to a multi-device positioning method, system and related apparatus. Background Art

[0002] Position estimation between multiple devices has always been the focus of researchers.

[0003] A positioning method between multiple devices is to determine a master device and multiple slave devices among the multiple devices. The master device sends a positioning signal, and the slave device provides signal feedback after receiving the positioning signal. Then, a coordinate system is established by connecting the two slave devices and the master device, and the coordinates of the slave device in the coordinate system are calculated using the signal feedback time and the signal propagation speed, so as to obtain the positional relationship between the three devices.

[0004] However, this positioning method has certain defects: at least three devices are required to complete the position determination between devices, and when the signal feedback time and signal propagation speed are used to determine the device position, the signal feedback time and propagation speed are easily affected by the environment, making the positioning result inaccurate. It can be seen that using this positioning method to estimate the position relationship between multiple devices is not ideal.

[0005] Therefore, how to better achieve positioning between multiple devices is a problem that needs to be solved urgently. Summary of the invention

[0006] The present application provides a multi-device positioning method, system and related apparatus, which realizes determining the positional relationship between multiple devices by using images collected by the devices.

[0007] In a first aspect, an embodiment of the present application provides a multi-device positioning method, the method comprising: obtaining visual features of a first image captured by a first device; obtaining visual features of a second image captured by a second device; the first image and the second image are images including a first object captured at different positions, the visual features of the first image describe the association between the first object and the pixel points in the first image, and the visual features of the second image describe the association between the first object and the pixel points in the second image; determining the positional relationship between the first device and the second device based on the visual features of the first image and the visual features of the second image.

[0008] It is understandable that the multi-device positioning method provided in the embodiment of the present application can be applied to the first device, the second device or other devices. In this way, the device can use the method to determine the position relationship between itself and other devices, and can also use the method to determine the position relationship between other multiple devices.

[0009] By implementing the method provided in the first aspect, the positional relationship between multiple devices can be determined using images captured by the devices. This method is not affected by the environment and can quickly and accurately determine the positional relationship between devices. Moreover, this method only requires two devices to quickly complete the positioning between devices, thereby expanding the application scope of the multi-device positioning method.

[0010] In combination with the first aspect, in a possible implementation, the visual features include one or more of the following: feature points, optical flow values, and transformation matrices; if the visual features of the first image and the second image include feature points, the positional relationship between the first device and the second device is determined by the relative camera pose of the first device and the second device, and the relative camera pose is determined by the feature points of the first object; if the visual features of the first image and the second image include optical flow values, the positional relationship between the first device and the second device is determined by the relative motion between the first device and the second device, and the relative motion is determined by the optical flow value of the first object; if the visual features of the first image and the second image include transformation matrices, the positional relationship between the first device and the second device is determined by transforming the positions of the first device and the second device to the same coordinate system using the transformation matrix of the first object extracted from the first image and the transformation matrix of the first object extracted from the second image.

[0011] That is to say, the method provided in the embodiment of the present application can combine one or more methods to calculate the position relationship between devices, thereby improving the flexibility of the multi-device positioning method.

[0012] In combination with the first aspect, in a possible implementation, the first object includes a first face, the visual features of the first image and the visual features of the second image include a transformation matrix of the first face, the transformation matrix of the first face extracted from the first image describes the relative position relationship between the first face and the ideal face model under the shooting perspective of the first device, and the transformation matrix of the first face extracted from the second image describes the relative position relationship between the first face and the ideal face model under the shooting perspective of the second device.

[0013] If the images collected by the devices contain the same face, the positional relationship between the devices can be determined based on the face in the image. This is because the features of the face are easier to identify, and using the face identified from the image to calculate the relative position between the devices can obtain a more accurate positioning result.

[0014] In combination with the first aspect, in a possible implementation method, the positional relationship between the first device and the second device is determined based on the visual features of the first image and the visual features of the second image, including: converting the coordinates of the first device in the world coordinate system corresponding to it to the coordinates in the local coordinate system corresponding to the second device based on the transformation matrix of the first face extracted from the first image; converting the coordinates of the first device in the local coordinate system to the coordinates in the world coordinate system corresponding to the second device based on the transformation matrix of the first face extracted from the second image; and determining the positional relationship between the first device and the second device based on the coordinates of the first device in the world coordinate system corresponding to the second device.

[0015] Among them, when using the face in the image to determine the positional relationship between devices, the transformation matrix of the face can be used to realize the coordinate transformation of the device between the world coordinate system and the local coordinate system, thereby achieving more accurate multi-device positioning.

[0016] In combination with the first aspect, in a possible implementation, the method also includes: obtaining visual features of a third image captured by a third device; the first image and the third image are images including a second object captured at different positions, the visual features of the first image also describe the association between the second object and the pixel points in the first image, and the visual features of the third image describe the association between the second object and the pixel points in the third image; determining the positional relationship between the first device and the third device based on the visual features of the first image and the visual features of the third image.

[0017] That is to say, the multi-device positioning method does not limit the number of devices, and can determine the positional relationship between two or more devices, and the images collected between different devices can contain different objects, for example, the first object can be different from the second object. In this way, when positioning more than three devices, these multiple devices do not need to shoot the same object, which can achieve more flexible multi-device positioning.

[0018] In combination with the first aspect, in a possible implementation, the method further includes: determining a positional relationship between the second device and the third device according to a positional relationship between the first device and the second device, and a positional relationship between the first device and the third device.

[0019] It can be seen that in addition to using the visual features of the images collected by the devices to calculate the positional relationship between devices, the positional relationship between devices that has been determined can also be used to deduce the positional relationship between other devices. In this way, the positional relationship between devices can be determined without using visual features for calculation, reducing the amount of calculation for multi-device positioning.

[0020] In combination with the first aspect, in a possible implementation manner, the positional relationship between the first device and the second device includes one or more of a relative height, a relative distance, a relative rotation angle, and a relative orientation between the first device and the second device.

[0021] In combination with the first aspect, in a possible implementation, after determining the positional relationship between the first device and the second device based on the visual features of the first image and the visual features of the second image, the method also includes: sending the positional relationship between the first device and the second device to the first device and / or the second device.

[0022] That is, after the position relationship between the devices is calculated, the position relationship can be sent to the positioning device so that the device can know the position relationship between itself and the surrounding devices.

[0023] In a second aspect, an embodiment of the present application provides an electronic device, characterized in that it includes a memory and one or more processors, the memory is used to store one or more programs; when the one or more processors execute the one or more programs, the electronic device implements the method described in the first aspect or any one of the implementation methods in the first aspect.

[0024] In conjunction with the second aspect, in a possible implementation, the electronic device may be the first device or the second device. In this way, the electronic device may determine the position relationship between itself and other devices using the method.

[0025] It is understandable that the electronic device may also be other devices besides the first device and the second device. In this way, the electronic device may determine the positional relationship between other devices using the method and learn the positional relationship between other devices.

[0026] In combination with the second aspect, in a possible implementation, the electronic device is a device selected from the first device and the second device based on any one of the following factors: the computing power of the device, the network identification of the device, the CPU idle rate of the device, and the need to obtain the relative position between devices.

[0027] In this way, the device most suitable for calculating multi-device positioning results can be selected from multiple devices according to certain strategies and different factors.

[0028] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on an electronic device, causes the electronic device to execute a method as described in the first aspect or any one of the implementations of the first aspect.

[0029] In a fourth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the method described in the first aspect or any one of the implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a scenario of a multi-device positioning method provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of the overall process of the multi-device positioning method provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of a communication system 1000 provided in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of the interaction between devices in the multi-device positioning method provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of the geometric relationship of the matching feature points provided in the embodiment of the present application;

[0035] Figure 6 A schematic diagram of the structure of a multi-device positioning system provided in an embodiment of the present application;

[0036] Figure 7 A schematic diagram of the hardware structure of an electronic device 400 provided in an embodiment of the present application;

[0037] Figure 8 A software structure block diagram of the electronic device 400 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0039] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0040] An embodiment of the present application provides a multi-device positioning method, which involves multiple devices. In the method, images captured by multiple devices of the same scene can be obtained, and the positional relationship between the multiple devices can be determined using the images captured by the multiple devices, thereby achieving multi-device positioning.

[0041] It can be seen that this method uses the differences and correlations between the images "seen" by different devices at different locations in the scene to estimate the positional relationship between the multiple devices. The positioning result is not affected by the environment, and the positional relationship between the devices can be obtained quickly and accurately. Only two devices are needed to complete the positioning between the devices, and the method has a wider range of applications.

[0042] Figure 1 A scenario diagram of a multi-device positioning method provided in an embodiment of the present application is shown.

[0043] like Figure 1 As shown, the scene includes a desk lamp. When determining the positional relationship between the electronic device 100 and the electronic device 200, the electronic device 100 and the electronic device 200 can respectively take images of the desk lamp in the scene, see image A and image B. By analyzing the same content captured in image A and image B, the shooting angles of the electronic device 100 and the electronic device 200 are analyzed, and the positional relationship between the electronic device 100 and the electronic device 200 is calculated. For example, Figure 1 In the illustrated scene, it can be calculated using image A and image B that the electronic device 100 is located on the left side of the electronic device 200 .

[0044] It can be seen that the multi-device positioning method only needs to use multiple devices to shoot the same scene, and the positional relationship between the multiple devices can be quickly and accurately calculated. In addition, the method does not require the device to be equipped with a binocular camera or an infrared camera to extract the depth features of the scene. The device only needs to be equipped with an ordinary camera, which has low hardware requirements for the device and improves the scalability of the multi-device positioning method.

[0045] It should be understood that Figure 1 The schematic diagram of the scenario is shown for illustrative purposes to facilitate understanding of the present solution and does not constitute a limitation on the embodiments of the present application. For example, more devices may be included in the scenario.

[0046] Combination Figure 1 In the scenario shown, the multi-device positioning method provided in the embodiment of the present application can control multiple devices in the scene to collect images of the scene, and then extract visual features of each of the multiple images. Afterwards, the positional relationship between the multiple devices is calculated using the visual features extracted from the multiple images.

[0047] Among them, the visual features reflect the association between the pixel points in the image and the spatial points in the real scene. Exemplarily, the visual features of the image can be extracted using a visual feature algorithm, wherein the visual feature algorithm may include: feature point method, optical flow method, pose estimation method, etc. Feature points are points where the grayscale value of the image changes dramatically or points with large curvature on the edge of the image. The optical flow value reflects the amount of motion of the object captured in the image, and the transformation matrix describes the relative position relationship between the object and the ideal model under the shooting angle of the device.

[0048] If the image does not contain a human face, the feature point method and / or the optical flow method may be used to extract the visual features of the image. When the feature point method is used to extract the visual features of the image, the visual features may refer to feature points in the image, and when the optical flow method is used to extract the visual features of the image, the visual features may refer to optical flow values.

[0049] If the image contains a face, the pose estimation method can be used to extract the visual features of the image. When the pose estimation method is used to extract the visual features of the image, the visual features can refer to the transformation matrix between the face captured in the image and the ideal face rigid model, which describes the relative position relationship between the face and the ideal face rigid model under the shooting angle of the electronic device.

[0050] When calculating the positional relationship between the multiple devices using visual features, the positional relationship between the two devices can be calculated using visual features extracted from images collected by each of the two devices.

[0051] The position relationship may include one or more of the following: relative orientation, relative rotation angle, relative distance, relative height, etc. Taking electronic device 100 and electronic device 100 as an example, the position relationship may mean that electronic device 100 is 5 meters away from electronic device 200, or may mean that electronic device 100 is located on the left side of electronic device 200, etc. The embodiment of the present application does not limit the presentation form of the position relationship.

[0052] In a specific implementation, the visual features of the same object contained in the images taken by the two devices can be used to calculate the positional relationship between the two devices. In this way, the positional relationship between the multiple devices can be estimated by using the differences and associations of the images "seen" by different devices at different locations in the scene.

[0053] Among them, for the images captured by two devices, if the visual features include feature points, the relative camera poses between the two devices can be calculated by matching the same feature points, that is, the feature points of the same object photographed by the two devices, so as to calculate the positional relationship between the two devices; if the visual features include optical flow values, the forward and backward movement from the shooting position of one of the devices to the shooting position of the other device can be calculated by the optical flow values ​​of the same object photographed by the two devices, so as to calculate the positional relationship between the two devices; if the visual features include transformation matrices, the transformation matrices of the same object photographed by the two devices can be used to perform coordinate transformation on the positions of the devices, and the camera positions of the two devices can be transformed into the same coordinate system to calculate the positional relationship between the two devices in the same coordinate system.

[0054] For example, Figure 1 Taking the scene shown in FIG. 1 as an example, the visual features extracted from image A and image B can be used to analyze the electronic device 100 and the electronic device 200 when taking the same object (e.g. Figure 1 The shooting angle of view when the electronic device 100 is shooting the table lamp shown in the scene) can be used to determine the relative position relationship between the two devices. Among them, it can be analyzed in combination with the image A shot by the electronic device 100 that the electronic device 100 is shooting toward the right front, thereby capturing the table lamp in the scene, and in combination with the image B shot by the electronic device 200 that the electronic device 200 is shooting toward the left front, thereby capturing the table lamp in the scene. In this way, it can be estimated that the electronic device 100 is located on the left side of the electronic device 200.

[0055] For a detailed description of the above-mentioned visual features, visual feature algorithms, and the use of visual features to calculate the positional relationship between devices, please refer to the subsequent detailed content.

[0056] Figure 2 The figure shows the overall process diagram of the multi-device positioning method provided in the embodiment of the present application.

[0057] like Figure 2 As shown, the method includes:

[0058] S101. Obtain images captured by multiple devices of the same scene.

[0059] The multiple devices may refer to two or more devices, and the multiple devices may be devices equipped with cameras, such as mobile phones, tablets, computers, etc. The embodiment of the present application does not limit the device types of the multiple devices.

[0060] S102. Determine the positional relationship between the multiple devices using the images captured by the multiple devices.

[0061] Exemplarily, the positional relationship between the multiple devices may be determined by extracting visual features in the image, wherein the visual features reflect the association between pixels in the image and spatial points in the real scene.

[0062] In a specific implementation, after acquiring images of the same scene captured by multiple devices, the visual features of the images can be extracted, and then the positional relationship between the two devices can be determined by analyzing the visual features of the images captured by two devices, thereby obtaining the positional relationship between any two devices among the multiple devices.

[0063] The positional relationship between the two devices may include the relative distance, relative height, relative rotation angle, relative orientation, etc. between the two devices.

[0064] Exemplarily, the positional relationship between devices can be determined using images through visual feature algorithms such as feature point method, optical flow method, pose estimation method, etc.

[0065] For detailed descriptions of the feature point method, optical flow method, and pose estimation method, please refer to the subsequent content, which will not be expanded here.

[0066] Figure 3 A schematic diagram of a communication system 1000 provided in an embodiment of the present application.

[0067] The communication system 1000 may include N devices, where N≥2. The N devices may include a master device and one or more slave devices. The N devices may be electronic devices such as mobile phones, tablets, computers, etc. Among them:

[0068] The slave device may be used to capture an image, extract visual features of the image, and send the visual features to the master device.

[0069] The master device can be used to obtain visual features extracted by the slave device, and use the visual features collected by multiple devices for the same scene to determine the positional relationship between the multiple devices, and send the positional relationship between the devices to the slave device. The multiple devices can refer to any two or more devices in the communication system 1000.

[0070] It is understandable that the slave device may also directly send the captured image to other devices, such as the master device, and the master device determines the visual features of the image. Preferably, the slave device may also send the visual features extracted from the image to the master device, so that the amount of data sent by the slave device can be reduced, while reducing the amount of calculation of the master device.

[0071] In addition, the master device may also collect images, so that when the master device locates multiple devices, the positional relationship between the master device and the slave device can be calculated using the visual features of the images collected by the master device and the visual features of the images collected by the slave devices. Alternatively, the master device may not collect images, so that the master device can only provide the calculation function in multi-device positioning, that is, the master device is only used to calculate the positional relationship between multiple slave devices.

[0072] For example, Figure 3 As shown, the communication system 1000 may include two devices: an electronic device 100 and an electronic device 200 .

[0073] The electronic device 100 may be a master device and the electronic device 200 may be a slave device, or the electronic device 100 may be a slave device and the electronic device 200 may be a master device.

[0074] Exemplarily, if electronic device 100 is a master device and electronic device 200 is a slave device, electronic device 100 and electronic device 200 can capture images of the same scene and extract visual features from the images. Electronic device 100 then aggregates the visual features extracted by electronic device 100 and electronic device 200, and uses the visual features extracted by electronic device 100 and electronic device 200 to calculate the positional relationship between electronic device 100 and electronic device 200, and finally sends the positional relationship to electronic device 200.

[0075] It is understandable that more devices may be included in the communication system 1000. The embodiment of the present application does not limit the number of devices included in the communication system 1000.

[0076] For example, the communication system 1000 may further include an electronic device 300, which may be a slave device different from the electronic device 200. In the communication system 1000, the electronic device 300 may also capture an image, extract visual features from the image, and send the extracted visual features to the electronic device 100. The electronic device 100 may calculate the positional relationship between the electronic device 100 and the electronic device 300 based on the visual features extracted by the electronic device 100 and the visual features extracted by the electronic device 300, and send the positional relationship between the electronic device 100 and the electronic device 300 to the electronic device 300.

[0077] Further optionally, the electronic device 100 may also calculate the positional relationship between the electronic device 200 and the electronic device 300 based on the visual features extracted by the electronic device 200 and the visual features extracted by the electronic device 300. Alternatively, preferably, the electronic device 100 may determine the positional relationship between the electronic device 200 and the electronic device 300 using the calculated positional relationship between the electronic device 100 and the electronic device 200, and the positional relationship between the electronic device 100 and the electronic device 300. In this way, the amount of calculation of the electronic device 100 may be reduced. Afterwards, the electronic device 100 sends the positional relationship between the electronic device 200 and the electronic device 300 to the electronic device 300.

[0078] Figure 4 A schematic diagram of the interaction between devices in the multi-device positioning method provided in an embodiment of the present application.

[0079] like Figure 4 As shown, the multi-device positioning method provided in the embodiment of the present application involves an electronic device 100 and an electronic device 200, and the interaction between the electronic device 100 and the electronic device 200 may include:

[0080] S201. The electronic device 100 captures an image A of the scene 1 through a camera.

[0081] The electronic device 100 may refer to an electronic device included in the communication system 1000 , and the communication system 1000 may also include other electronic devices, such as the electronic device 200 .

[0082] The electronic device 100 may trigger the execution of step S201 in the following two situations:

[0083] 1) When the electronic device 100 needs to obtain the position relationship with other devices, it triggers the execution of step S201

[0084] Exemplarily, the other device may refer to other electronic devices in the communication system 1000 except the electronic device 100 , such as the electronic device 200 .

[0085] Taking the other device as electronic device 200 as an example, electronic device 100 can subscribe to the location awareness capability through the application when the application needs to obtain the position relationship between electronic device 100 and electronic device 200, so that electronic device 100 can trigger the execution of step S201 by using the location awareness capability.

[0086] Among them, the access perception capability can be used to enable an electronic device to start a camera to capture images, process the images, extract visual features of the images, and obtain the positional relationship between devices, wherein the positional relationship between devices is calculated using the visual features of images captured by multiple devices.

[0087] 2) When the electronic device 100 needs to obtain the position relationship between the electronic device 100 and the other device, the electronic device 100 triggers the execution of step S201

[0088] In this case, before the electronic device 100 executes step S201, it may obtain a positioning request sent by other devices, such as the electronic device 200, where the positioning request is used to request positioning between devices, triggering the electronic device 100 to execute step S201.

[0089] In addition, in the communication system 1000, the electronic device 100 can establish a communication connection with other electronic devices. These devices that establish a communication connection are devices that need to perform inter-device positioning.

[0090] Specifically, the communication connection may be a wired connection or a wireless connection. The wireless connection may be a short-distance connection such as a high-fidelity wireless communication (wireless fidelity, Wi-Fi) connection, a Bluetooth connection, an infrared connection, an NFC connection, a ZigBee connection, or a long-distance connection, and the long-distance connection includes but is not limited to a long-distance connection based on 2G, 3G, 4G, 5G and subsequent standard protocols of mobile networks.

[0091] The electronic device 100 can realize data exchange with other electronic devices in the communication system 1000 through the communication connection, such as sending visual features extracted from image A, or receiving visual features extracted from images captured by other electronic devices, receiving or sending positional relationships between devices, etc.

[0092] S202. The electronic device 100 extracts visual features of image A.

[0093] Visual features reflect the correspondence between pixels in an image and spatial points in a real scene, and can be used to calculate the positional relationship between devices.

[0094] Exemplarily, the electronic device 100 may extract the visual features of the image A by any one or more of the following methods:

[0095] 1) Feature point method

[0096] The feature point method extracts feature points from an image and determines the camera position by matching feature points in different images.

[0097] In the embodiments of the present application, it can be assumed that the relative position relationship between the camera and the device itself is fixed, and the feature points in the images captured by different devices can be used to analyze the camera postures when the different devices capture images. The camera posture of the camera can be regarded as the posture of the device, and the positional relationship between the devices can be calculated.

[0098] Among them, feature points refer to points where the grayscale value of the image changes dramatically or points with large curvature on the edge of the image (i.e., the intersection of two edges). Commonly used feature points include: corner points, edge points, spots, etc. Feature points can reflect the essential characteristics of the image and identify the target object in the image.

[0099] The visual features extracted using the feature point method may refer to feature points in the image.

[0100] In some embodiments, feature points in an image may be extracted through image texture information, image region grayscale statistics, or through frequency change, wavelet transform, and other transformations followed by spatial transformation. For example, feature points in an image may be extracted through a scale-invariant feature transform (SIFT) algorithm, a speeded up robust features (SURF) algorithm, or a feature detection (Oriented FAST and RotatedBRIEF, ORB) algorithm.

[0101] 2) Optical flow method

[0102] The optical flow method uses the changes in pixels in the image sequence in the time domain and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby determining the movement of the camera that captured the image sequence.

[0103] In an embodiment of the present application, two devices that capture images of the same scene can be regarded as a moving device in motion, and the devices before and after the motion. The images captured by the two devices on the same scene can be regarded as the upper and lower frames of images in an image sequence. The optical flow method can be used to calculate the movement from the shooting perspective of one device to the shooting perspective of another device, thereby determining the positional relationship between the two devices.

[0104] The visual features extracted using the optical flow method may refer to optical flow values ​​extracted from an image, and the optical flow values ​​reflect motion information of an object.

[0105] 3) Pose estimation method

[0106] The pose estimation method is to identify the object in the image, calculate the transformation relationship between the object and the imaginary object model at the shooting angle of the device, and thus estimate the pose of the object at the shooting angle of the device.

[0107] In an embodiment of the present application, a common object within the shooting field of view can be found between devices, and the position of the object under the shooting angles of different devices can be estimated to deduce the relationship between the shooting angles of the devices, thereby obtaining the position relationship between the devices.

[0108] Exemplarily, in an embodiment of the present application, the positional relationship between devices can be calculated by estimating the position and posture of the characters contained in the scene. Preferably, the positional relationship between devices can be calculated by identifying the faces of the characters in the scene and taking the faces as the object of position and posture estimation. This is because the features of the face are easier to identify and distinguish, and it is more accurate to calculate the positional relationship between devices using the face identified from the image, and the preset ideal face rigid body model can be used to estimate the position and posture of the face, without the need for additional modeling, saving the cost of modeling.

[0109] In some implementations, if scene 1 includes a person, a pose estimation method may be used to extract feature points of the person's face in the image, and the feature points of the person's face may be used to calculate the positional relationship between devices.

[0110] Among them, if scene 1 contains people, taking the model-based pose estimation method as an example, after the electronic device 100 captures image A for scene 1, feature points of the face (for example, the first face) can be extracted from image A, and then an ideal face rigid body model in a real scene is used to match the feature points of the face extracted from image A with the feature points on the ideal face rigid body model, and calculate the transformation matrix between the face in scene 1 and the ideal face, so that the transformation matrix is ​​used as the visual feature extracted from image A. The transformation matrix includes a translation matrix and a rotation matrix corresponding to the face and the ideal face rigid body model under the shooting perspective of the electronic device 100, which describes the relative position relationship between the face and the ideal face rigid body model under the shooting perspective of the electronic device 100.

[0111] That is to say, if scene 1 contains a person, a model-based pose estimation method can be used to calculate the positional relationship between devices. When a model-based pose estimation method is used to calculate the positional relationship between devices, the visual features extracted from the image can refer to the transformation matrix of the face of the person contained in scene 1, which describes the relative pose of the face of the person in scene 1 and the ideal face under the shooting angle when the image is taken.

[0112] In order to better understand the principle of extracting the transformation matrix of an image using a pose estimation algorithm, the detailed process of extracting the transformation matrix of an image by the electronic device 100 is shown below.

[0113] Step 1: Identify facial features of the face in image A.

[0114] After the electronic device 100 captures the image A, it can perform face detection on the image A to identify whether the image A contains facial features of a human face.

[0115] For example, facial features of a person in image A may be identified using an image feature extraction algorithm based on histogram of oriented gradient (HOG) or a face recognition algorithm such as Fast R-CNN.

[0116] Step 2: Extract facial feature points from facial features.

[0117] Among them, the facial feature points in the facial features of the human face can be marked by the facial key point algorithm, including multiple (for example, 68) feature points such as the tip of the nose, the corners of the eyes, the corners of the mouth, etc.

[0118] Step 3: Determine the transformation matrix between the facial feature points and the ideal facial rigid body model.

[0119] The ideal rigid body model of a human face is an imaginary three-dimensional model in real space.

[0120] Exemplarily, the transformation matrix between the facial feature points and the ideal facial rigid body model can be calculated by the following formula 1:

[0121] R*(X,Y,Z)+T=(x,y,z) Formula 1

[0122] Among them, the transformation matrix includes: rotation matrix R and translation matrix T, (x, y, z) is the coordinates of the facial feature point p in the image in the camera coordinate system of the electronic device 100, and (X, Y, Z) is the spatial point P on the ideal facial rigid body model that matches the facial feature point p.

[0123] In some implementations, the transformation matrix corresponding to the image may be extracted using a PnP (Perspective-n-Point) algorithm.

[0124] It is understandable that visual features in an image may be extracted by other methods, and the embodiments of the present application are not limited thereto. In some embodiments, the visual features in an image may be extracted by combining the above-mentioned methods. For example, if the electronic device 100 uses both the feature point method and the pose estimation method to obtain visual features in image A, the visual features extracted from image A may include: feature points and a transformation matrix.

[0125] In addition, in the embodiment of the present application, in addition to the electronic device 100 extracting the visual features of the image A, the electronic device 100 may also send the image A to other devices, and the other devices may extract the visual features of the image A. The other devices may refer to the main device or to a device with a stronger computing capability in the communication system 1000. The embodiment of the present application does not limit the device for extracting the visual features of the image.

[0126] S203. The electronic device 200 collects image B of scene 1 through a camera.

[0127] The electronic device 200 may refer to an electronic device included in the communication system 1000 .

[0128] The electronic device 200 may trigger the execution of step S203 in the following two situations:

[0129] 1) When the electronic device 200 needs to obtain the position relationship with other devices, it triggers the execution of step S203

[0130] Exemplarily, the other device may refer to other electronic devices in the communication system 1000 except the electronic device 200 , such as the electronic device 100 .

[0131] Taking the other device as electronic device 100 as an example, electronic device 200 can subscribe to the location awareness capability through the application when the application needs to obtain the position relationship between electronic device 200 and electronic device 100, so that electronic device 100 can trigger the execution of step S203 by using the location awareness capability.

[0132] 2) When the electronic device 200 needs to obtain the position relationship between the electronic device 200 and other devices, the electronic device 200 triggers the execution of step S203

[0133] In this case, before the electronic device 200 executes step S203, it may obtain a positioning request sent by other devices, such as the electronic device 100, where the positioning request is used to request positioning between devices, triggering the electronic device 200 to execute step S203.

[0134] It is understandable that the relevant description in the image B captured by the electronic device 200 and the relevant description in the image A captured by the electronic device 100, and the specific contents not mentioned in step S203 can refer to the relevant description in the aforementioned step S201, which will not be repeated here.

[0135] In some embodiments, in order to ensure that the electronic device 100 and the electronic device 200 can capture images containing the same object, the electronic device 100 and the electronic device 200 can display a shooting preview interface when shooting images, so that the user can view the currently shot scene, and the user can guide different devices to shoot the same object. Furthermore, during the shooting process, one of the devices can also specify an object that needs to be captured in the image, and send the specified object to other devices, so that the images captured by multiple devices all contain the specified object. In this way, the user does not need to view multiple devices at the same time and coordinate the shooting angles of multiple devices, which facilitates the user's operation. The embodiments of the present application do not limit the implementation method of multiple devices shooting the same object.

[0136] S204. The electronic device 200 extracts visual features of image B.

[0137] Exemplarily, the electronic device 200 may extract visual features of the image B by any one or more of the following methods: 1) feature point method; 2) optical flow method; 3) pose estimation method.

[0138] For detailed descriptions of these three methods, please refer to the relevant content in the aforementioned step S202, which will not be repeated here.

[0139] The electronic devices in the communication system 1000, such as the electronic device 100 and the electronic device 200, may use the same method to extract visual features of an image.

[0140] In addition, a master device may be selected among the devices in the communication system 1000. The master device is used to aggregate visual features extracted by multiple devices in the communication system 1000 and calculate the positional relationship between the multiple devices using the visual features extracted by the multiple devices.

[0141] Exemplarily, the master device may be determined in any of the following ways:

[0142] 1) Randomly select a device as the master device

[0143] In the communication system 1000, a device may be randomly selected from multiple devices as a master device. In this way, randomly selecting a master device can speed up the decision-making process and select the master device as soon as possible for calculating the position relationship between devices.

[0144] 2) Determine the master device based on the device's computing power

[0145] Since the master device needs to aggregate and calculate the data sent by multiple devices and determine the positional relationship between the devices, the master device needs to have good computing capabilities.

[0146] In the communication system 1000, the device with the strongest computing capability can be determined as the master device among the multiple devices, so as to increase the computing speed of calculating the positioning of multiple devices as much as possible.

[0147] 3) Select the master device from the devices that need to obtain the position relationship between the devices

[0148] Since the master device is used to calculate the positional relationship between devices, the device that needs to obtain the positional relationship between devices is determined as the master device, and the communication step of sending the positional relationship between devices to other devices can be omitted, so that the device that needs to obtain the positional relationship between devices can obtain the positional relationship between devices at the first time.

[0149] 4) Determine the master device based on the device’s central processing unit (CPU) idle rate

[0150] The CPU idle rate reflects the system's busyness to a certain extent.

[0151] In the communication system 1000, the device with the highest CPU idle rate can be determined as the master device, so that the computation efficiency of multi-device positioning between computing devices can be improved as much as possible.

[0152] It is understandable that the master device may also be determined by other means, and the embodiments of the present application are not limited to this.

[0153] In some embodiments, the master device may be selected based on the device identifier. Exemplarily, the identifier may refer to a network identifier (Network ID). In a specific implementation, the master device may be selected from the network identifiers of multiple devices in lexicographic order. For example, if there are three devices with network identifiers A, B, and C, A is the first letter in lexicographic order, and the device with network identifier A may be selected as the master device.

[0154] The selection of a master device from the communication system 1000 may be triggered in any of the following situations:

[0155] 1) After the communication system 1000 is established, or after a device in the communication system 1000 is updated, the selection of the master device is triggered

[0156] That is to say, after the communication system 1000 is established, the main device can be selected from the multiple devices included in the communication system 1000, or, if there are updates such as the addition and removal of devices in the communication system 1000, the main device can be reselected from the updated multiple devices.

[0157] 2) When a device in the communication system 1000 needs to obtain the position relationship with other devices, it triggers the selection of the master device

[0158] That is to say, before calculating the positional relationship between devices through the master device each time, the master device can be reselected and the positional relationship between devices can be calculated using the reselected master device. In this way, the device that uses visual features to calculate the positional relationship between devices can be different each time.

[0159] It is understandable that in the embodiment of the present application, in addition to the electronic device 200 extracting the visual features of the image B, the electronic device 200 may also send the image B to other devices, and the other devices may extract the visual features of the image B. Among them, the other devices may refer to the main device, or may refer to the device with stronger computing power in the communication system 1000, and the embodiment of the present application does not limit the device for extracting the visual features of the image.

[0160] The following steps S205 - S207 take the electronic device 100 as an example to describe the interaction process between the electronic device 100 and the electronic device 200 .

[0161] S205. The electronic device 200 sends the visual features extracted from the image B to the electronic device 100.

[0162] Accordingly, the electronic device 100 obtains the visual features extracted from the image B sent by the electronic device 200.

[0163] It is understandable that if the communication system 1000 also includes other devices, such as the electronic device 300, the electronic device 100 can also obtain the visual features sent by the other devices.

[0164] Exemplarily, the electronic device 200 may send the visual features extracted from the image B to the electronic device 100 via the distributed soft bus. The embodiment of the present application does not limit the communication method used by the electronic device 200 to send the visual features.

[0165] S206. The electronic device 100 determines the positional relationship between the two devices based on the visual features extracted from the two devices.

[0166] The electronic device 100 may determine the positional relationship between the electronic device 100 and the electronic device 200 based on the visual features extracted from the electronic device 100 and the electronic device 200 .

[0167] Depending on the visual features, the positional relationship between devices can be determined by one or more of the following methods:

[0168] 1) Feature point method

[0169] If the visual features extracted by the device include feature points, the electronic device 100 can determine the positional relationship between the devices by using the principle of epipolar geometry. The positional relationship between the devices can be obtained by matching the feature points in the two images and calculating the relative camera poses of the two devices using the matched feature points. The matched feature points are feature points of the same object contained in the images taken by the two devices.

[0170] In order to better understand the principle of using the feature point method to determine the positional relationship between devices, the following describes in detail the process of using the feature point method to determine the positional relationship between electronic devices 100 and 200, taking the calculation of the positional relationship between electronic device 100 and electronic device 200 as an example.

[0171] Step 1: Match the feature points extracted from image A and image B.

[0172] If the same content in the scene is captured in the two images, then the two images contain the same feature points. Matching the feature points of image A and image B is to determine the same feature points in image A and image B.

[0173] For example, key point matching may be performed using a fast library for approximate nearest neighbors (FLANN) algorithm.

[0174] Step 2: Calculate the relative camera pose of the two cameras through the matched feature points.

[0175] Among them, the relative camera pose of the two cameras describes the motion relationship between the relative translation and relative rotation of the two cameras.

[0176] Figure 5 A schematic diagram of the geometric relationship of the matching feature points provided in an embodiment of the present application.

[0177] like Figure 5 As shown, the feature point p1 in image A matches the feature point p2 in image B, O1 is the camera center of the device (electronic device 100) that shoots image A, O2 is the camera center of the device (electronic device 200) that shoots image B, P is the spatial point corresponding to the feature points P1 and P2 in the real scene, the pole e1 is the intersection of the baseline O1O2 and the imaging plane where image A is located, the pole e2 is the intersection of the baseline O1O2 and the imaging plane where image B is located, the epipolar line l1 is the intersection of the epipolar plane O1O2P and the imaging plane where image A is located, and the epipolar line l2 is the intersection of the epipolar plane O1O2P and the imaging plane where image B is located.

[0178] Combination Figure 5The geometric relationship between the feature points p1 and p2 shown in the figure, the feature points p1 and p2 have the constraint relationship of the following formula 2:

[0179]

[0180] E=T^R,F=K -T EK -1

[0181]

[0182] Among them, p1 is the feature point in image A, p2 is the feature point in image B, p1 and p2 are a pair of matching feature points, x1 and x2 are the coordinates of P on the normalized planes of the two cameras respectively, K is the camera parameter in the camera, F is the basic matrix, E is the essential matrix, and the relative camera pose of the two cameras includes: rotation matrix R and translation matrix t.

[0183] For example, 8 pairs of matching feature points in image A and image B can be found, the basic matrix F in the above formula 2 is solved by the eight-point method, and then the essential matrix E in the above formula 2 is solved by the singular value decomposition (SVD), and finally the essential matrix E is used to solve the relative camera pose transformation R, t of the two cameras.

[0184] Step 3: Determine the positional relationship between the electronic device 100 and the electronic device 200 using the relative camera postures of the two cameras.

[0185] Assuming that the relative positions of the camera and the device are fixed, the relative camera pose of the two cameras is the relative camera pose of the two devices. In this way, the relative rotation angle of the electronic device 100 and the electronic device 200 can be determined by the rotation matrix R, and the relative orientation, relative distance, relative height, etc. of the electronic device 100 and the electronic device 200 can be determined by the translation matrix t.

[0186] In some implementations, the positional relationship between devices may be estimated using a structure from motion (SFM) algorithm.

[0187] 2) Optical flow method

[0188] If the visual features extracted by the device include optical flow values, the forward and backward movement of the spatial point on the two images can be determined by the optical flow values, and the positional relationship between the two devices corresponding to the two images can be determined based on the movement.

[0189] The optical flow values ​​of the same object captured in the two images may be used to determine the positional relationship between the two devices.

[0190] 3) Pose estimation method

[0191] If the visual features extracted by the device include a transformation matrix, the transformation matrix can be used to determine the positional relationship between the devices by transforming the coordinates of the photographed face, where the face is an object included in the images taken by both devices.

[0192] In order to better understand the principle of determining the positional relationship between devices using the pose estimation method, the following takes the calculation of the positional relationship between the electronic device 100 and the electronic device 200 as an example to describe in detail the detailed process of determining the positional relationship between the electronic device 100 and the electronic device 200 using the transformation matrix extracted from the images respectively collected by the electronic device 100 and the electronic device 200:

[0193] Step 1: The coordinates M of the electronic device 100 in its corresponding world coordinate system are converted to the coordinates N in the local coordinate system of the electronic device 200 through the transformation matrix extracted from the image captured by the electronic device 100 .

[0194] The world coordinate system is a coordinate system of real space established with a certain point in the real scene as the coordinate origin. The world coordinate system corresponding to the electronic device 100 may refer to a coordinate system established with the location of the electronic device 100 as the coordinate origin.

[0195] The local coordinate system is a coordinate system where the object is located, with the center of the object as the coordinate origin. Since the electronic device 100 and the electronic device 200 use the same ideal human face rigid body model, the local coordinate system corresponding to the electronic device 200 is the same as the local coordinate system corresponding to the electronic device 100.

[0196] Step 2: Using the transformation matrix extracted from the image captured by the electronic device 200 , the coordinate N of the electronic device 100 in the local coordinate system is converted to the coordinate L in the world coordinate system corresponding to the electronic device 200 .

[0197] The world coordinate system corresponding to the electronic device 200 may refer to a coordinate system established with the location of the electronic device 200 as the coordinate origin.

[0198] The coordinate L reflects the relative positional relationship between the electronic device 100 and the electronic device 200 in the same coordinate system, that is, the world coordinate system corresponding to the electronic device 200 .

[0199] Step 3: Use the coordinate L to determine the positional relationship between the electronic device 100 and the electronic device 200 .

[0200] In the world coordinate system where the electronic device 200 is located, the coordinate origin can be regarded as the position where the electronic device 200 is located, and the coordinate L can be regarded as the position where the electronic device 100 is located.

[0201] Exemplarily, the coordinate L can be compared with the origin of the world coordinate system where the electronic device 200 is located to determine the relative orientation, relative distance, relative height and other positional relationships between the electronic device 100 and the electronic device 200; based on the initial angle in the world coordinate system where the electronic device 200 is located, the orientation of the electronic device 100 at the coordinate L is determined as the relative rotation angle between the electronic device 100 and the electronic device 200.

[0202] For example, if the values ​​of each coordinate axis in the coordinate L are greater than 0, and the coordinate origin is taken as the position of the electronic device 200 , it can be seen that the electronic device 100 is located in front of the right side of the electronic device 200 .

[0203] The above steps 1-3 illustrate the process of transforming the coordinates of the electronic device 100 into the world coordinate system where the electronic device 200 is located to determine the positional relationship between the electronic device 100 and the electronic device 200. In some other implementations, the coordinates of the electronic device 200 may also be transformed into the world coordinate system where the electronic device 100 is located to determine the positional relationship between the electronic device 100 and the electronic device 200. The embodiments of the present application do not limit this.

[0204] It is understandable that the electronic device 100 can combine the above-mentioned multiple methods to calculate the position relationship between devices, and use the position relationships calculated by multiple methods to calibrate the calculation results, so that the position relationship calculated by the electronic device 100 is more accurate.

[0205] In addition, if the communication system 1000 includes other devices in addition to the electronic device 100 and the electronic device 200, the electronic device 100 can calculate the position relationship between the two devices in any of the following ways:

[0206] 1) The electronic device 100 uses the visual features extracted by the two devices to calculate the positional relationship between the two devices

[0207] In this way, even if the electronic device 100 calculates the positional relationship between other devices incorrectly, it will not affect the calculation of the positional relationship between the two devices.

[0208] 2) The electronic device 100 uses the positional relationship between the two devices and the same device to calculate the positional relationship between the two devices

[0209] Exemplarily, assuming that the communication system 1000 includes electronic devices 100, 200, and 300, the image captured by electronic device 100 is image A, the image captured by electronic device 200 is image B, and the image captured by electronic device 300 is image C, electronic device 100 can use the visual features of the same object captured in image A and image B to determine the positional relationship between electronic device 100 and electronic device 200, and use the visual features of the same object captured in image A and image C to determine the positions of electronic device 100 and electronic device 300.

[0210] Furthermore, the electronic device 100 can calculate the positional relationship between the electronic device 200 and the electronic device 300 by using the positional relationship between the electronic device 100 and the electronic device 200 and the electronic device 300. For example, if the electronic device 100 calculates that the electronic device 200 is on the left side of the electronic device 100 and the electronic device 300 is in front of the electronic device 100, then the electronic device 200 is behind the left side of the electronic device 300.

[0211] In this way, for the positional relationship between some devices in the communication system 1000, there is no need to use visual features to calculate, and the positional relationship between the devices that has been calculated can be directly used for estimation, thereby reducing the amount of calculation.

[0212] In some embodiments, if a master device, such as electronic device 100, needs to calculate the positional relationship between any two devices among all devices in the communication system 1000, the master device can first use the visual features of the master device and all slave devices to calculate the positional relationship between the master device and each slave device, and then use the positional relationship between the master device and each slave device to infer the positional relationship between the slave devices.

[0213] In addition, after the electronic device 100 calculates the positional relationship between devices, the electronic device 100 can sense the positional relationship between the surrounding electronic devices and the electronic device 100. Applications in the electronic device 100 that subscribe to the location perception capability can use the positional relationship between the devices to perform corresponding functions, such as window switching between multiple devices, application dragging, etc., to enhance the user experience in multi-device scenarios.

[0214] S207 . The electronic device 100 sends the positional relationship between the devices to the electronic device 200 .

[0215] Correspondingly, the electronic device 200 receives the positional relationship between devices sent by the electronic device 100 .

[0216] The electronic device 100 may send all the calculated location information to the electronic device 200 , or may send only the location information related to the location of the electronic device 200 , or may send the location information required by the electronic device 200 .

[0217] It is understandable that if the electronic device 200 does not need to use the positional relationship between devices, step S207 is an optional step.

[0218] In this way, after the electronic device 200 obtains the position relationship between devices, the electronic device 200 can perceive the position relationship between the surrounding electronic devices and the electronic device 200. The application in the electronic device 200 that subscribes to the location perception capability can use the position relationship between the devices to perform corresponding functions, such as window switching between multiple devices, application dragging, etc., to enhance the user experience in the multi-device scenario.

[0219] In addition, if the communication system 1000 also includes other devices, the electronic device 100 may also send the calculated positional relationship between the devices to the other devices.

[0220] In the embodiment of the present application, the electronic device 100 may also be referred to as a first device, the electronic device 200 may also be referred to as a second device, the electronic device 300 may also be referred to as a third device, the image A may also be referred to as a first image, the image B may also be referred to as a second image, and the image C may also be referred to as a third image.

[0221] In general, the positioning method provided in the embodiment of the present application uses images collected by the device to perform positioning between devices, has fast calculation speed, low latency, is not affected by the environment, and can provide real-time and accurate device positioning results for the application.

[0222] Figure 6 A schematic diagram of the structure of a multi-device positioning system provided in an embodiment of the present application.

[0223] The multi-device positioning system involves an electronic device 100 and an electronic device 200 .

[0224] like Figure 6 As shown, both electronic device 100 and electronic device 200 include a capability subscription module, which is used to subscribe to the orientation perception capability. The orientation perception capability can use a visual feature algorithm, which is used to process images captured by the camera and extract visual features, or, further, use visual features to calculate the positional relationship between devices.

[0225] Among them, the capability subscription module is used to enable the application to complete the call to the orientation perception capability, so that the application can obtain the position relationship between devices by calling the orientation perception capability.

[0226] The orientation perception capability can use the visual feature algorithm to obtain visual features using images captured by the camera; and, when the device is a slave device, the obtained visual features are sent to the master device through the distributed soft bus, and the positional relationship between devices calculated by the master device is obtained through the distributed soft bus; when the device is a master device, the visual features obtained by each device are received through the distributed soft bus, and the positional relationship between devices is calculated using the visual features obtained by each device, and the positional relationship between devices is sent to each device through the distributed soft bus.

[0227] Understandably, Figure 6 The structural diagram of the multi-device positioning system is described by taking the multi-device positioning system including the electronic device 100 and the electronic device 200 as an example. In other embodiments of the present application, the multi-device positioning system may also include more electronic devices, which is not limited in the embodiments of the present application.

[0228] Figure 7 A schematic diagram of the hardware structure of the electronic device 400 is shown.

[0229] The electronic device 400 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device.

[0230] The electronic device 400 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0231] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 400. In other embodiments of the present application, the electronic device 400 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0232] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0233] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0234] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0235] The charging management module 140 is used to receive charging input from a charger. While the charging management module 140 is charging the battery 142 , it can also power the electronic device through the power management module 141 .

[0236] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160.

[0237] The wireless communication function of the electronic device 400 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0238] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 400 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0239] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 400. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0240] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0241] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 400. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0242] In some embodiments, the antenna 1 of the electronic device 400 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 400 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0243] The electronic device 400 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0244] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0245] The electronic device 400 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0246] The ISP is used to process data fed back by the camera 193. For example, when taking a photo, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye.

[0247] The camera 193 is used to capture still images or videos. In some embodiments, the electronic device 400 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0248] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 400 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0249] Video codecs are used to compress or decompress digital videos. The electronic device 400 may support one or more video codecs. Thus, the electronic device 400 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0250] NPU is a neural network (NN) computing processor, which can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. NPU can realize applications such as intelligent cognition of electronic device 400, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0251] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0252] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0253] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0254] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 400. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function.

[0255] The electronic device 400 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0256] In the embodiment of the present application, if the electronic device 400 is a main device, for example Figure 4 The electronic device 100 shown in FIG.

[0257] The camera 193 can be used to capture images of the scene;

[0258] The processor 110 may be used to obtain visual features of an image captured by the camera 193, and calculate a positional relationship between devices based on the visual features acquired by each device;

[0259] The mobile communication module 150 or the wireless communication module 160 may be used to receive visual features sent from a slave device, and to send positional relationships between devices to the slave device;

[0260] The internal memory 121 may be used to store the visual features obtained by each device and the positional relationship between the devices.

[0261] In the embodiment of the present application, if the electronic device 400 is a slave device, for example Figure 4 The electronic device 200 shown:

[0262] The camera 193 can be used to capture images of the scene;

[0263] The processor 110 may be used to obtain visual features of an image based on an image captured by the camera 193;

[0264] The mobile communication module 150 or the wireless communication module 160 may be used to send visual features to the host device, and receive the positional relationship between the devices sent by the host device;

[0265] The internal memory 121 may be used to store the obtained visual features and the positional relationship between devices.

[0266] The electronic device may be a portable terminal device equipped with iOS, Android, Microsoft or other operating systems, such as a mobile phone, a tablet computer, a wearable device, etc., and may also be a non-portable terminal device such as a laptop computer with a touch-sensitive surface or a touch panel, a desktop computer with a touch-sensitive surface or a touch panel. The software system of the electronic device 400 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 400.

[0267] Figure 8 This is a software structure block diagram of the electronic device 400 according to an embodiment of the present application.

[0268] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0269] The application layer can include a series of application packages.

[0270] like Figure 8 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0271] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0272] like Figure 8 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0273] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0274] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0275] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0276] The phone manager is used to provide communication functions of the electronic device 400, such as management of call status (including connecting, hanging up, etc.).

[0277] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0278] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0279] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0280] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0281] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0282] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0283] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0284] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0285] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0286] A 2D graphics engine is a drawing engine for 2D drawings.

[0287] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0288] The following is an illustrative description of the workflow of the software and hardware of the electronic device 400 in conjunction with capturing a photo scene.

[0289] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0290] It should be understood that each step in the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0291] The present application also provides an electronic device, which may include: a memory and a processor. The memory may be used to store a computer program; the processor may be used to call the computer program in the memory, so that the electronic device executes the method executed by the electronic device 100 or the electronic device 200 in any of the above embodiments.

[0292] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method performed by the electronic device 100 or the electronic device 200 in any of the above embodiments.

[0293] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0294] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0295] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0296] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0297] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0298] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute a method executed by any one of the electronic devices 100 or 200 in any of the above embodiments.

[0299] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by any one of the electronic device 100 or the electronic device 200 in any of the above embodiments.

[0300] It should be understood that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (digitalsignal processor, DSP), an application specific integrated circuit (AP 800plication specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined to perform. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0301] In addition, an embodiment of the present application further provides a device. The device may be a component or a module, and the device may include one or more processors and a memory connected to each other. The memory is used to store a computer program. When the computer program is executed by one or more processors, the device executes the method in each of the above method embodiments.

[0302] Among them, the device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present application are used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0303] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0304] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0305] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0306] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-device positioning method, characterized in that: The method comprises: Acquire visual features of a first image captured by a first device; Acquire visual features of a second image captured by a second device; the first image and the second image are images including a first object captured at different positions, the visual features of the first image describe an association between the first object and pixels in the first image, and the visual features of the second image describe an association between the first object and pixels in the second image; The positional relationship between the first device and the second device is determined according to the visual features of the first image and the visual features of the second image.

2. The method according to claim 1, characterized in that The visual features include one or more of the following: feature points, optical flow values, and transformation matrices; If the visual features of the first image and the second image include feature points, the positional relationship between the first device and the second device is determined by a relative camera pose of the first device and the second device, and the relative camera pose is determined by the feature points of the first object; If the visual features of the first image and the second image include optical flow values, the positional relationship between the first device and the second device is determined by a relative motion amount between the first device and the second device, and the relative motion amount is determined by the optical flow value of the first object; If the visual features of the first image and the second image include transformation matrices, the positional relationship between the first device and the second device is determined by transforming the positions of the first device and the second device into the same coordinate system using the transformation matrix of the first object extracted from the first image and the transformation matrix of the first object extracted from the second image.

3. The method according to claim 1 or 2, characterized in that: The first object includes a first human face, the visual features of the first image and the visual features of the second image include a transformation matrix of the first human face, the transformation matrix of the first human face extracted from the first image describes the relative position relationship between the first human face and an ideal human face model under the shooting perspective of the first device, and the transformation matrix of the first human face extracted from the second image describes the relative position relationship between the first human face and the ideal human face model under the shooting perspective of the second device.

4. The method according to claim 3, characterized in that Determining a positional relationship between the first device and the second device according to a visual feature of the first image and a visual feature of the second image includes: According to the transformation matrix of the first face extracted from the first image, the coordinates of the first device in the world coordinate system corresponding to the first device are converted into coordinates in the local coordinate system corresponding to the second device; Converting coordinates of the first device in a local coordinate system to coordinates of the second device in a world coordinate system according to the transformation matrix of the first face extracted from the second image; The positional relationship between the first device and the second device is determined according to the coordinates of the first device in the world coordinate system corresponding to the second device.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire visual features of a third image captured by a third device; the first image and the third image are images including a second object captured at different positions, the visual features of the first image further describe an association between the second object and pixels in the first image, and the visual features of the third image describe an association between the second object and pixels in the third image; The positional relationship between the first device and the third device is determined according to the visual features of the first image and the visual features of the third image.

6. The method according to claim 5, characterized in that The method further comprises: The positional relationship between the second device and the third device is determined according to the positional relationship between the first device and the second device and the positional relationship between the first device and the third device.

7. The method according to any one of claims 1 to 6, characterized in that: The positional relationship between the first device and the second device includes one or more of a relative height, a relative distance, a relative rotation angle, and a relative orientation between the first device and the second device.

8. The method according to any one of claims 1 to 7, characterized in that: After determining the positional relationship between the first device and the second device according to the visual features of the first image and the visual features of the second image, the method further includes: The position relationship between the first device and the second device is sent to the first device and / or the second device.

9. An electronic device, characterized in that: The electronic device comprises a memory and one or more processors, wherein the memory is used to store one or more programs; when the one or more processors execute the one or more programs, the electronic device implements the method as claimed in any one of claims 1 to 8.

10. The method according to claim 9, characterized in that The electronic device is the first device or the second device.

11. The method according to claim 10, characterized in that The electronic device is a device selected from the first device and the second device based on any one of the following factors: computing capability of the device, network identification of the device, CPU idle rate of the device, and a need to obtain relative positions between devices.

12. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 8.

Images