Camera link determination method, apparatus, and electronic device for ar glasses

Abstract

The application provides a camera link determination method and device of an AR glasses and an electronic device. The method comprises the following steps: obtaining a plurality of first relative pose relationships, each first relative pose relationship being a relative pose relationship between two adjacent cameras, and the first relative pose relationship comprising a position relationship and an orientation vector relationship of the cameras at a current time; constructing a spanning tree comprising the plurality of cameras; calculating the weight of each edge of the spanning tree according to the first relative pose relationship, and determining that a minimum spanning tree formed by edges with the minimum weight is the camera link at the current time. The method determines the pose information of the cameras by obtaining the images of the cameras, constructs the spanning tree of the cameras, and determines that the minimum spanning tree formed by the edges with the minimum weight is the camera link, thereby solving the problem that the fixed link of the cameras in the AR glasses in the prior art causes the inter-frame matching of the cameras at some time to be less, and the problem that the stability of the AR glasses is low.

Description

Technical Field

[0001] This application relates to the field of AR glasses, and more specifically, to a method, apparatus, computer-readable storage medium, processor, and electronic device for determining camera links in AR glasses. Background Technology

[0002] SLAM (simultaneous localization and mapping) is widely used in autonomous driving and augmented reality. SLAM technology mainly includes two tasks: scene localization and map reconstruction, that is, locating the current camera's pose information while building an environmental map. Visual SLAM is needed for navigation in AR glasses.

[0003] In AR glasses, the usual practice is to match cameras sequentially according to a fixed camera link. However, this camera link strategy can cause the link to break down if a camera in the middle of the link is blocked or malfunctions, making it impossible to use the effective information from other cameras. Therefore, the lack of inter-frame matching in AR glasses leads to lower stability in the operation of AR glasses.

[0004] Therefore, there is an urgent need for a solution to address the problem of low stability in AR glasses operation caused by insufficient inter-frame matching in existing AR glasses cameras.

[0005] The information disclosed above in the background section is only intended to enhance the understanding of the background art of the art described herein. Therefore, the background art may contain certain information that does not constitute prior art known to those skilled in the art in this country. Summary of the Invention

[0006] The main objective of this application is to provide a method, apparatus, computer-readable storage medium, processor, and electronic device for determining camera links in AR glasses, in order to solve the problem of low stability in AR glasses operation caused by insufficient inter-frame matching in the prior art.

[0007] To achieve the above objectives, according to one aspect of this application, a method for determining camera links in AR glasses is provided. The AR glasses include multiple cameras. The method includes: acquiring multiple first relative pose relationships, each first relative pose relationship being the relative pose relationship between two adjacent cameras, the first relative pose relationship including the position relationship and orientation vector relationship of the cameras at the current moment; constructing a spanning tree containing the multiple cameras; calculating the weight of each edge of the spanning tree based at least on the first relative pose relationships, and determining the minimum spanning tree formed by the edge with the smallest weight as the camera link at the current moment.

[0008] Further, calculating the weight of each edge of the spanning tree based at least on the first relative pose relationship includes: calculating the distance between the projections of the optical centers of the two cameras corresponding to each edge of the spanning tree onto a predetermined plane, wherein the normal vector of the predetermined plane is the angle bisector of the two camera orientation vectors; and determining the weight of the edge based on the distance, wherein the weight is proportional to the distance.

[0009] Further, calculating the weights of each edge of the spanning tree based at least on the first relative pose relationship also includes: determining whether there is an abnormal camera among the plurality of cameras, wherein the abnormal camera is a faulty camera and / or an occluded camera; if the abnormal camera exists among the two cameras corresponding to the edge, determining that the weight of the edge corresponding to the abnormal camera is infinite; if the abnormal camera does not exist among the two cameras corresponding to the edge, calculating the distance between the projections of the optical centers of the two cameras corresponding to the edge onto the predetermined plane; and determining the weight of the edge based on the distance, wherein the weight is proportional to the distance.

[0010] Further, determining whether an abnormal camera exists among the plurality of cameras includes: acquiring images of the plurality of cameras at the current moment to obtain a plurality of first images; dividing the first images into a plurality of image regions and calculating the grayscale value of each image region; determining whether the difference in the grayscale values ​​of the image regions corresponding to the same position in the first images of two adjacent cameras is within a first predetermined range; if the difference is within the first predetermined range, determining that no abnormal camera exists; if the difference is not within the first predetermined range, determining that an abnormal camera exists among two adjacent cameras.

[0011] Further, when it is determined that there is an abnormal camera among two adjacent cameras, determining the abnormal camera among the two adjacent cameras includes: determining the abnormal camera among the two adjacent cameras based on two gray values ​​corresponding to the difference between the two adjacent cameras not being within the first predetermined range; and if there is a gray value that is not within the second predetermined range, determining the camera in the image region corresponding to the gray value that is not within the second predetermined range as the abnormal camera.

[0012] Furthermore, the method further includes: determining the connection edge between the camera link at the previous moment and the camera link at the current moment, to obtain a camera link including the camera link at the previous moment, the connection edge, and the camera link at the current moment, wherein the connection edge is the edge with the smallest weight connecting the camera link at the previous moment and the camera link at the current moment, and the previous moment is the moment when the previous frame image was captured before the current moment.

[0013] Further, determining the connection edge between the camera link at the previous moment and the camera link at the current moment includes: obtaining a plurality of second relative pose relationships between the cameras at the current moment and the previous moment, each second relative pose relationship being the relative pose relationship between the cameras at the current moment and the previous moment; calculating the weights of a plurality of candidate edges based at least on the second relative pose relationships, the candidate edges being any edge connecting the camera link at the previous moment and the camera link at the current moment; and determining the candidate edge with the smallest weight as the connection edge.

[0014] According to another aspect of this application, a camera link determination device for AR glasses is also provided. The AR glasses include multiple cameras. The device includes an acquisition unit, a construction unit, and a first determination unit. The acquisition unit is used to acquire multiple first relative pose relationships, each of which is the relative pose relationship between two adjacent cameras. The first relative pose relationship includes the position relationship and orientation vector relationship of the cameras at the current moment. The construction unit is used to construct a spanning tree containing the multiple cameras. The first determination unit is used to calculate the weight of each edge of the spanning tree based at least on the first relative pose relationships, and determine the minimum spanning tree formed by the edge with the smallest weight as the camera link at the current moment.

[0015] According to another aspect of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed by a processor, it controls the device where the computer storage medium is located to perform any of the described AR glasses camera link determination methods.

[0016] According to another aspect of this application, a processor is also provided for running a computer program, wherein the computer program, when running, executes any of the described AR glasses camera link determination methods.

[0017] According to another aspect of this application, an electronic device is also provided, including a camera link determination device for AR glasses, the camera link determination device for AR glasses being used to perform any of the camera link determination methods for AR glasses described above.

[0018] Applying the technical solution of this application, in the camera link determination method for AR glasses, the AR glasses include multiple cameras, and the method includes: acquiring multiple first relative pose relationships, each first relative pose relationship being the relative pose relationship between two adjacent cameras, the first relative pose relationship including the position relationship and orientation vector relationship of the cameras at the current moment; constructing a spanning tree containing the multiple cameras; calculating the weight of each edge of the spanning tree based at least on the first relative pose relationships, the spanning tree formed by the edge with the smallest weight is the minimum spanning tree, and determining the minimum spanning tree as the camera link at the current moment. This application determines the weights of camera edges by obtaining the first relative pose relationship of the cameras in AR glasses, and then constructs a camera spanning tree. The minimum spanning tree formed by the edge with the smallest weight is determined as the camera link. The smaller the weight, the higher the similarity between the images of the two corresponding cameras, and the higher the matching between camera frames. Therefore, the camera link formed by the edge with the smallest weight is the link with the most common observations. This method constructs camera links at each current time, that is, the method updates camera links in real time. The higher the similarity between the images of the two cameras connected by the edge of the camera link, the better. This avoids the problem in the prior art where the fixed links of cameras in AR glasses lead to less matching between camera frames at certain times, resulting in low stability of AR glasses operation. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 A flowchart of a camera link determination method for AR glasses according to an embodiment of this application is shown;

[0021] Figure 2 A schematic diagram showing the projection of two cameras of AR glasses according to an embodiment of this application onto a predetermined plane is shown;

[0022] Figure 3 A complete link diagram of the camera of AR glasses according to an embodiment of this application is shown;

[0023] Figure 4 A camera link diagram of AR glasses according to an embodiment of this application is shown;

[0024] Figure 5 This illustrates the camera link between adjacent moments in AR glasses according to existing technology;

[0025] Figure 6 The camera link of AR glasses at adjacent moments according to an embodiment of this application is shown;

[0026] Figure 7 This is a schematic diagram of a camera link determination device for AR glasses according to an embodiment of this application.

[0027] The above figures include the following reference numerals:

[0028] 101. First camera; 102. Second camera; 103. Third camera; 104. Fourth camera; 105. Fifth camera; 106. Sixth camera; 107. Predetermined plane; 108. Camera link at the current moment; 109. Camera link at the previous moment; 110. Connecting edge. Detailed Implementation

[0029] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0030] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0031] It should be understood that when an element (such as a layer, film, region, or substrate) is described as being "on" another element, the element may be directly on the other element, or there may be an intermediate element present. Furthermore, in the specification and claims, when an element is described as being "connected" to another element, the element may be "directly connected" to the other element, or "connected" to the other element via a third element.

[0032] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0033] Edges: In AR glasses, visual SLAM is required for navigation, which usually requires calculating common features between different cameras, including points, lines, objects, etc. This relationship between two cameras is called an "edge".

[0034] Link: Multiple edges constitute a "link".

[0035] As described in the background section, the low stability of AR glasses operation is caused by insufficient inter-frame matching in the existing AR glasses camera. In order to solve the above problems, this application provides a method, apparatus, computer-readable storage medium, processor and electronic device for determining the camera link of AR glasses.

[0036] According to a typical embodiment of this application, a method for determining the camera link of AR glasses is provided, wherein the AR glasses include multiple cameras.

[0037] Figure 1 This is a flowchart of a camera link determination method for AR glasses according to an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0038] Step S101: Obtain multiple first relative pose relationships, each of which is the relative pose relationship between two adjacent cameras. The first relative pose relationship includes the position relationship and orientation vector relationship of the cameras at the current moment.

[0039] Step S102: Construct a spanning tree containing multiple of the above-mentioned cameras;

[0040] Step S103: Calculate the weights of each edge of the above-mentioned spanning tree based at least on the first relative pose relationship, and determine the minimum spanning tree formed by the edge with the smallest weight as the camera link at the current moment.

[0041] In the above-mentioned AR glasses camera link determination method, firstly, multiple first relative pose relationships are obtained, each of which is the relative pose relationship between two adjacent cameras. The first relative pose relationship includes the position relationship and orientation vector relationship of the cameras at the current moment. Then, a spanning tree containing multiple cameras is constructed. Finally, the weight of each edge of the spanning tree is calculated based on at least the first relative pose relationships, and the minimum spanning tree formed by the edge with the smallest weight is determined as the camera link at the current moment. This method determines the weights of camera edges by obtaining the first relative pose relationship of the cameras in AR glasses, and then constructs a camera spanning tree. The minimum spanning tree formed by the edges with the smallest weights is determined as the camera link. The smaller the weight, the higher the similarity between the images of the two corresponding cameras, and the higher the matching between camera frames. Therefore, the camera link formed by the edges with the smallest weights is the link with the most common observations. This method constructs camera links at each current moment, that is, the method updates camera links in real time. The higher the similarity between the images of the two cameras connected by the edges of the camera link, the better. This avoids the problem in the prior art where the fixed links of cameras in AR glasses lead to less matching between camera frames at certain moments, resulting in low stability of AR glasses operation.

[0042] In practical applications, the aforementioned relative pose relationship can be acquired using motion sensors, and then corrected for accuracy using visual SLAM technology. Visual SLAM technology infers the camera's pose in an unknown environment based on captured images and simultaneously constructs an environment map; its basic principle is multi-view geometry. The goal of visual SLAM is to simultaneously recover the camera motion parameters and the 3D structure of the scene corresponding to each frame of the image, where each camera motion parameter includes the camera's position and orientation vectors.

[0043] In order to efficiently and accurately calculate the weights of camera edges, thereby more accurately determining the camera links at the current moment, i.e., obtaining more camera links for inter-frame matching, and thus obtaining more stable AR glasses, in one embodiment of this application, the weights of each edge of the above-mentioned spanning tree are calculated based on at least the first relative pose relationship, including: calculating the distance between the projections of the optical centers of the two cameras corresponding to each edge of the above-mentioned spanning tree on a predetermined plane, wherein the normal vector of the predetermined plane is the angle bisector of the two camera orientation vectors; and determining the weights of the edges based on the distances, wherein the weights are proportional to the distances.

[0044] In practical applications, using the above weight calculation method, even if AR glasses contain 10 cameras, the computational load of the calculated camera links is relatively small and the efficiency is high. The time to calculate the effective camera links can even be as low as microseconds.

[0045] Specifically, Figure 2 This is a schematic diagram of the projection of two cameras of AR glasses onto a predetermined plane. The first camera 101 and the second camera 102 are two cameras with the same intrinsic parameters. The projections (without considering distortion) of the light emitted by the first camera 101 and the second camera 102 onto the predetermined plane 107 are ABCD and EFGH, respectively. The normal vector of the predetermined plane 107 is... Let l be the angle bisector of the orientation vectors of the first camera 101 and the second camera 102. The projections of the optical centers of the first camera 101 and the second camera 102 onto the predetermined plane 107 are P and Q, respectively, and the distance between P and Q is l. The weight of the edge between the two cameras is based on the similarity of the images obtained from the two cameras. The higher the similarity, the smaller the weight; the lower the similarity, the larger the weight. Therefore, the closer P and Q are, the larger the overlap between ABCD and EFGH, meaning that the images obtained through the first camera 101 and the second camera 102 are more similar. Thus, using the distance l between the projections of the optical centers of the first camera 101 and the second camera 102 onto the predetermined plane 107 to measure the similarity of the cameras can efficiently obtain the weight of the edge between the cameras, and thus quickly obtain the camera links.

[0046] In practical applications, the relationship between cameras is represented by Tbci, where b represents the common body coordinate system, which is the reference coordinate system. ci is the i-th camera. Constructing a spanning tree containing multiple of the above cameras yields the complete camera link, as shown below. Figure 3 As shown, Figure 3 This is the complete link diagram of the six cameras: camera 101, camera 102, camera 103, camera 104, camera 105, and camera 106. The weights of the edges between camera 101 and camera 102, camera 103 and camera 104, and camera 105 and camera 106 are all 0.0223063. The weights of the edges between camera 101 and camera 103, camera 103 and camera 105, camera 102 and camera 104, and camera 106 are also 0.0223063. The weight of the edge between camera 4 and the sixth camera 106 is 0.0027083. The weights of the edges between camera 101 and camera 104, camera 102 and camera 103, camera 103 and camera 106, and camera 104 and camera 105 are all 0.0225538. The weights of the edges between camera 101 and camera 105, and between camera 102 and camera 106 are all 0.00467191. The minimum spanning tree formed by the edges with the smallest weights is determined to be the camera link. Figure 4 As shown.

[0047] In another embodiment of this application, calculating the weights of each edge of the spanning tree based at least on the first relative pose relationship further includes: determining whether there is an abnormal camera among the plurality of cameras, wherein the abnormal camera is a faulty camera and / or an occluded camera; if there is an abnormal camera among the two cameras corresponding to the edge, determining that the weight of the edge corresponding to the abnormal camera is infinite; if there is no abnormal camera among the two cameras corresponding to the edge, calculating the distance between the projections of the optical centers of the two cameras corresponding to the edge onto the predetermined plane; and determining the weight of the edge based on the distance, wherein the weight is proportional to the distance. The presence of an abnormal camera in AR glasses can cause the camera link to become invalid. By identifying the abnormal camera and setting its weight to infinity, it can be ensured that there is no abnormal camera in the camera link subsequently determined by comparing weights. This ensures that effective feature extraction can be performed between the cameras connected in the camera link, and improves extraction efficiency. In other words, even if some cameras are occluded or the cameras are faulty, the AR glasses can still operate normally, further improving the stability of the AR glasses.

[0048] To quickly determine whether there is a malfunction or obstruction in the camera, in another embodiment of this application, determining whether there is an abnormal camera among the plurality of cameras includes: acquiring images of the plurality of cameras at the current moment to obtain a plurality of first images; dividing the first images into a plurality of image regions and calculating the grayscale value of each of the image regions; determining whether the difference in the grayscale values ​​of the image regions corresponding to the same position in the first images of two adjacent cameras is within a first predetermined range; if the difference is within the first predetermined range, determining that there is no abnormal camera; if the difference is not within the first predetermined range, determining that there is an abnormal camera among the two adjacent cameras.

[0049] Specifically, the above grayscale can be calculated using, but is not limited to, the method for calculating the average grayscale.

[0050] To further accurately identify abnormal cameras, in another embodiment of this application, when it is determined that there is an abnormal camera among two adjacent cameras, identifying the abnormal camera among two adjacent cameras includes: determining the abnormal camera among two adjacent cameras based on two gray values ​​corresponding to the difference between the two adjacent cameras not being within the first predetermined range; and if there is a gray value that is not within the second predetermined range, determining the camera in the image region corresponding to the gray value that is not within the second predetermined range as the abnormal camera.

[0051] In another embodiment of this application, the method further includes: determining the connecting edge between the camera link at the previous moment and the camera link at the current moment, to obtain a camera link including the camera link at the previous moment, the connecting edge, and the camera link at the current moment, wherein the connecting edge is the edge with the smallest weight connecting the camera link at the previous moment and the camera link at the current moment, and the previous moment is the moment when the previous frame image was captured before the current moment. In this embodiment, by connecting camera links at adjacent moments, and since the camera link at the previous moment is also a camera link composed of the edge with the smallest weight at the previous moment, and the connecting edge is also the edge with the smallest weight connecting the two camera links, the resulting multiple camera links at multiple moments are also camera links with the smallest weight edges, further ensuring that the common observation of the obtained multiple connected camera links at multiple moments is maximized.

[0052] In order to quickly obtain the relative pose relationship of the cameras, thereby efficiently obtaining the weights of the camera edges and efficiently and accurately determining the connection edges, in another embodiment of this application, determining the connection edge between the camera link at the previous moment and the camera link at the current moment includes: obtaining a plurality of second relative pose relationships between the cameras at the current moment and the previous moment, each of the second relative pose relationships being the relative pose relationship between the cameras at the current moment and the previous moment; calculating the weights of a plurality of candidate edges based at least on the second relative pose relationships, the candidate edges being any of the edges connecting the camera link at the previous moment and the camera link at the current moment; and determining the candidate edge with the smallest weight as the connection edge.

[0053] In practical applications, Figure 5 This refers to the camera links of adjacent moments in existing AR glasses technology, including the camera link 108 at the current moment, the camera link 109 at the previous moment, and the connecting edge 110. The AR glasses contain two cameras, a first camera 101 and a second camera 102. Using the method of this application, the weight of the candidate edge is determined by calculating the distance between the projections of the optical centers of each pair of cameras onto a predetermined plane. The projection weight of the optical center of the second camera 102 on the predetermined plane is smaller between two adjacent moments, and the candidate edge between the two second cameras 102 at two moments is the connecting edge mentioned above. The camera links of adjacent moments in the AR glasses are as follows: Figure 6 As shown.

[0054] This application also provides a camera link determination device for AR glasses. It should be noted that the camera link determination device for AR glasses in this application can be used to execute the camera link determination method for AR glasses provided in this application. The following describes the camera link determination device for AR glasses provided in this application.

[0055] Figure 7 This is a schematic diagram of a camera link determination device for AR glasses according to an embodiment of this application. Figure 7 As shown, the device includes:

[0056] The acquisition unit 10 is used to acquire multiple first relative pose relationships, each of which is the relative pose relationship between two adjacent cameras. The first relative pose relationship includes the position relationship and orientation vector relationship of the cameras at the current moment.

[0057] Construction unit 20 is used to construct a spanning tree containing multiple of the above-mentioned cameras;

[0058] The first determining unit 30 is used to calculate the weight of each edge of the above-mentioned spanning tree based at least on the above-mentioned first relative pose relationship, and to determine the minimum spanning tree formed by the edge with the smallest weight as the camera link at the above-mentioned current moment.

[0059] In the aforementioned camera link determination device for AR glasses, firstly, an acquisition unit acquires multiple first relative pose relationships, each of which is the relative pose relationship between two adjacent cameras. The first relative pose relationship includes the position relationship and orientation vector relationship of the cameras at the current moment. Then, a construction unit constructs a spanning tree containing multiple cameras. Finally, a first determination unit calculates the weight of each edge of the spanning tree based at least on the first relative pose relationships, and determines the minimum spanning tree formed by the edge with the smallest weight as the camera link at the current moment. This device determines the weights of camera edges by acquiring the first relative pose relationship of the cameras in the AR glasses, and then constructs a camera spanning tree. The minimum spanning tree formed by the edges with the smallest weights is determined as the camera link. The smaller the weight, the higher the similarity between the images of the two corresponding cameras, and the higher the matching between camera frames. Therefore, the camera link formed by the edges with the smallest weights is the link with the most common observations. The device constructs camera links at each current moment, that is, the device updates camera links in real time. The higher the similarity between the images of the two cameras connected by the edges of the camera link, the better. This avoids the problem in the prior art where the fixed links of cameras in AR glasses lead to less matching between camera frames at certain times, resulting in lower stability of AR glasses operation.

[0060] In practical applications, the aforementioned relative pose relationship can be acquired using motion sensors, and then corrected for accuracy using visual SLAM technology. Visual SLAM technology infers the camera's pose in an unknown environment based on captured images and simultaneously constructs an environment map; its basic principle is multi-view geometry. The goal of visual SLAM is to simultaneously recover the camera motion parameters and the 3D structure of the scene corresponding to each frame of the image, where each camera motion parameter includes the camera's position and orientation vectors.

[0061] In order to efficiently and accurately calculate the weights of camera edges, thereby more accurately determining the camera links at the current moment, i.e., obtaining more camera links for inter-frame matching, and thus obtaining more stable AR glasses, in one embodiment of this application, the first determining unit includes a first calculation module and a first determining module. The first calculation module is used to calculate the distance between the projections of the optical centers of the two cameras corresponding to each edge of the spanning tree onto a predetermined plane, wherein the normal vector of the predetermined plane is the angle bisector of the two camera orientation vectors; the first determining module is used to determine the weights of the edges based on the distances, wherein the weights are proportional to the distances.

[0062] In practical applications, using the above weight calculation method, even if AR glasses contain 10 cameras, the computational load of the calculated camera links is relatively small and the efficiency is high. The time to calculate the effective camera links can even be as low as microseconds.

[0063] Specifically, Figure 2 This is a schematic diagram of the projection of two cameras of AR glasses onto a predetermined plane. The first camera 101 and the second camera 102 are two cameras with the same intrinsic parameters. The projections (without considering distortion) of the light emitted by the first camera 101 and the second camera 102 onto the predetermined plane 107 are ABCD and EFGH, respectively. The normal vector of the predetermined plane 107 is... Let l be the angle bisector of the orientation vectors of the first camera 101 and the second camera 102. The projections of the optical centers of the first camera 101 and the second camera 102 onto the predetermined plane 107 are P and Q, respectively, and the distance between P and Q is l. The weight of the edge between the two cameras is based on the similarity of the images obtained from the two cameras. The higher the similarity, the smaller the weight; the lower the similarity, the larger the weight. Therefore, the closer P and Q are, the larger the overlap between ABCD and EFGH, meaning that the images obtained through the first camera 101 and the second camera 102 are more similar. Thus, using the distance l between the projections of the optical centers of the first camera 101 and the second camera 102 onto the predetermined plane 107 to measure the similarity of the cameras can efficiently obtain the weight of the edge between the cameras, and thus quickly obtain the camera links.

[0064] In practical applications, the relationship between cameras is represented by Tbci, where b represents the common body coordinate system, which is the reference coordinate system. ci is the i-th camera. Constructing a spanning tree containing multiple of the above cameras yields the complete camera link, as shown below. Figure 3 As shown, Figure 3This is the complete link diagram of the six cameras: camera 101, camera 102, camera 103, camera 104, camera 105, and camera 106. The weights of the edges between camera 101 and camera 102, camera 103 and camera 104, and camera 105 and camera 106 are all 0.0223063. The weights of the edges between camera 101 and camera 103, camera 103 and camera 105, camera 102 and camera 104, and camera 106 are also 0.0223063. The weight of the edge between camera 4 and the sixth camera 106 is 0.0027083. The weights of the edges between camera 101 and camera 104, camera 102 and camera 103, camera 103 and camera 106, and camera 104 and camera 105 are all 0.0225538. The weights of the edges between camera 101 and camera 105, and between camera 102 and camera 106 are all 0.00467191. The minimum spanning tree formed by the edges with the smallest weights is determined to be the camera link. Figure 4 As shown.

[0065] In another embodiment of this application, the first determining unit further includes a second determining module, a third determining module, a second calculating module, and a fourth determining module. The second determining module determines whether an abnormal camera exists among the plurality of cameras, wherein the abnormal camera is a faulty camera and / or an occluded camera. The third determining module determines that the weight of the edge corresponding to the abnormal camera is infinite if an abnormal camera exists among the two cameras corresponding to the edge. The second calculating module calculates the distance between the projections of the optical centers of the two cameras corresponding to the edge onto the predetermined plane if no abnormal camera exists among the two cameras corresponding to the edge. The fourth determining module determines the weight of the edge based on the distance, and the weight is proportional to the distance. The presence of an abnormal camera in AR glasses can cause the camera link to become invalid. By identifying the abnormal camera and setting its weight to infinity, it is ensured that no abnormal camera exists during the subsequent determination of the camera link by comparing weights. This ensures that effective feature extraction can be performed between the cameras connected in the camera link, and improves extraction efficiency. In other words, even if some cameras are occluded or the cameras are faulty, the AR glasses can still operate normally, further improving the stability of the AR glasses.

[0066] To quickly determine whether a camera is malfunctioning or obstructed, in another embodiment of this application, the second determining module includes an acquisition submodule, a calculation submodule, a first determining submodule, a second determining submodule, and a third determining submodule. The acquisition submodule acquires multiple images of the cameras at the current moment to obtain multiple first images. The calculation submodule divides the first images into multiple image regions and calculates the grayscale value of each image region. The first determining submodule determines whether the difference in grayscale values ​​of the image regions corresponding to the same position in the first images of two adjacent cameras is within a first predetermined range. The second determining submodule determines that no abnormal camera exists if the difference is within the first predetermined range. The third determining submodule determines that an abnormal camera exists in one of the two adjacent cameras if the difference is not within the first predetermined range.

[0067] Specifically, the above grayscale can be calculated using, but is not limited to, the method for calculating the average grayscale.

[0068] To further accurately identify abnormal cameras, in another embodiment of this application, the third determining submodule further includes determining the abnormal camera among two adjacent cameras based on two gray values ​​corresponding to the difference between the two adjacent cameras not falling within the first predetermined range, and determining the camera in the image region corresponding to the gray value not falling within the second predetermined range as the abnormal camera if one of the two gray values ​​is not falling within the second predetermined range.

[0069] In another embodiment of this application, the method further includes a second determining unit, which is used to determine the connecting edge between the camera link at the previous moment and the camera link at the current moment, thereby obtaining a camera link including the camera link at the previous moment, the connecting edge, and the camera link at the current moment. The connecting edge is the edge with the smallest weight connecting the camera link at the previous moment and the camera link at the current moment. The previous moment is the moment when the previous frame image was captured before the current moment. In this embodiment, by connecting the camera links at adjacent moments, and since the camera link at the previous moment is also a camera link composed of the edge with the smallest weight at the previous moment, and the connecting edge is also the edge with the smallest weight connecting the two camera links, the resulting multiple camera links at multiple moments are also camera links with the smallest weight edges, further ensuring that the common observation of the obtained multiple connected camera links at multiple moments is maximized.

[0070] In order to quickly obtain the relative pose relationships of the cameras, thereby efficiently obtaining the weights of the camera edges and efficiently and accurately determining the connecting edges, in another embodiment of this application, the second determining unit includes an acquisition module, a third calculation module, and a fifth determining module. The acquisition module is used to acquire the second relative pose relationships of multiple cameras at the current time and the previous time, and each of the second relative pose relationships is the relative pose relationship of the camera at the current time and the previous time. The third calculation module is used to calculate the weights of multiple candidate edges based at least on the first relative pose relationships. The candidate edges are any of the edges connecting the camera link at the previous time and the camera link at the current time. The fifth determining module is used to determine the candidate edge with the smallest weight as the connecting edge.

[0071] In practical applications, Figure 5 This refers to the camera links of adjacent moments in existing AR glasses technology, including the camera link 108 at the current moment, the camera link 109 at the previous moment, and the connecting edge 110. The AR glasses contain two cameras, a first camera 101 and a second camera 102. Using the method of this application, the weight of the candidate edge is determined by calculating the distance between the projections of the optical centers of each pair of cameras onto a predetermined plane. The projection weight of the optical center of the second camera 102 on the predetermined plane is smaller between two adjacent moments, and the candidate edge between the two second cameras 102 at two moments is the connecting edge mentioned above. The camera links of adjacent moments in the AR glasses are as follows: Figure 6 As shown.

[0072] The camera link determination device for the AR glasses mentioned above includes a processor and a memory. The acquisition unit, the first determination unit, the construction unit, and the second determination unit are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.

[0073] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the issue of low stability in AR glasses due to insufficient inter-frame matching in existing technologies.

[0074] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0075] This invention provides a computer-readable storage medium including a stored computer program, wherein when the computer program is run by a processor, it controls the device where the computer storage medium is located to execute any of the aforementioned AR glasses camera link determination methods.

[0076] This invention provides a processor for running a program, wherein the program executes the camera link determination method for the AR glasses.

[0077] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0078] Step S101: Obtain multiple first relative pose relationships, each of which is the relative pose relationship between two adjacent cameras. The first relative pose relationship includes the position relationship and orientation vector relationship of the cameras at the current moment.

[0079] Step S102: Construct a spanning tree containing multiple of the above-mentioned cameras;

[0080] Step S103: Calculate the weights of each edge of the above-mentioned spanning tree based at least on the first relative pose relationship, and determine the minimum spanning tree formed by the edge with the smallest weight as the camera link at the current moment.

[0081] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0082] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0083] Step S101: Obtain multiple first relative pose relationships, each of which is the relative pose relationship between two adjacent cameras. The first relative pose relationship includes the current position relationship and orientation vector relationship of the cameras.

[0084] Step S102: Construct a spanning tree containing multiple of the above-mentioned cameras;

[0085] Step S103: Calculate the weights of each edge of the above-mentioned spanning tree based at least on the first relative pose relationship, and determine the minimum spanning tree formed by the edge with the smallest weight as the camera link at the current moment.

[0086] In another typical embodiment of this application, an electronic device is also provided, including a camera link determination device for AR glasses, wherein the camera link determination device for AR glasses is used in any of the aforementioned camera link determination methods for AR glasses.

[0087] The aforementioned electronic device includes a camera link determination device for AR glasses. This device determines the weights of camera edges by acquiring the first relative pose relationship of the cameras in the AR glasses, and then constructs a camera spanning tree. The minimum spanning tree formed by the edges with the smallest weights is determined as the camera link. The smaller the weight, the higher the similarity between the images of the two corresponding cameras, and the higher the matching between camera frames. Therefore, the camera link formed by the edges with the smallest weights is the link with the most common observations. This device constructs camera links at each current moment, that is, the device updates camera links in real time. The higher the similarity between the images of the two cameras connected by the edges of the camera link, the better. This avoids the problem in the prior art where the fixed links of cameras in AR glasses lead to less matching between camera frames at certain times, resulting in lower stability of AR glasses operation.

[0088] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0089] 1) In the camera link determination method of AR glasses in this application, firstly, multiple first relative pose relationships are obtained, each of the first relative pose relationships being the relative pose relationship between two adjacent cameras, the first relative pose relationship including the position relationship and orientation vector relationship of the cameras at the current moment; then, a spanning tree containing multiple cameras is constructed; finally, at least according to the first relative pose relationships, the weight of each edge of the spanning tree is calculated, and the minimum spanning tree formed by the edge with the smallest weight is determined as the camera link at the current moment. This method determines the weights of camera edges by obtaining the first relative pose relationship of the cameras in AR glasses, and then constructs a camera spanning tree. The minimum spanning tree formed by the edges with the smallest weights is determined as the camera link. The smaller the weight, the higher the similarity between the images of the two corresponding cameras, and the higher the matching between camera frames. Therefore, the camera link formed by the edges with the smallest weights is the link with the most common observations. This method constructs camera links at each current moment, that is, the method updates camera links in real time. The higher the similarity between the images of the two cameras connected by the edges of the camera link, the better. This avoids the problem in the prior art where the fixed links of cameras in AR glasses lead to less matching between camera frames at certain moments, resulting in low stability of AR glasses operation.

[0090] 2) In the camera link determination device for AR glasses of this application, firstly, a plurality of first relative pose relationships are obtained by the acquisition unit, each of the first relative pose relationships being the relative pose relationship between two adjacent cameras, the first relative pose relationship including the position relationship and orientation vector relationship of the cameras at the current moment; then, a spanning tree containing the plurality of cameras is constructed by the construction unit; finally, the weight of each edge of the spanning tree is calculated by the first determination unit at least according to the first relative pose relationships, and the minimum spanning tree formed by the edge with the smallest weight is determined as the camera link at the current moment. This device determines the weights of camera edges by acquiring the first relative pose relationship of the cameras in the AR glasses, and then constructs a camera spanning tree. The minimum spanning tree formed by the edges with the smallest weights is determined as the camera link. The smaller the weight, the higher the similarity between the images of the two corresponding cameras, and the higher the matching between camera frames. Therefore, the camera link formed by the edges with the smallest weights is the link with the most common observations. The device constructs camera links at each current moment, that is, the device updates camera links in real time. The higher the similarity between the images of the two cameras connected by the edges of the camera link, the better. This avoids the problem in the prior art where the fixed links of cameras in AR glasses lead to less matching between camera frames at certain times, resulting in lower stability of AR glasses operation.

[0091] 3) The electronic device of this application includes a camera link determination device for AR glasses. This device determines the weight of the camera edges by obtaining the first relative pose relationship of the cameras of the AR glasses, and then constructs the camera's spanning tree. The minimum spanning tree formed by the edge with the smallest weight is determined as the camera link. The smaller the weight, the higher the similarity between the images of the two corresponding cameras, and the higher the matching between camera frames. Therefore, the camera link formed by the edge with the smallest weight is the link with the most common observations. This device constructs the camera link at each current moment, that is, the device updates the camera link in real time. The higher the similarity between the images of the two cameras connected by the edge of the camera link, the better. This avoids the problem in the prior art where the fixed links of cameras in AR glasses lead to less matching between camera frames at certain times, resulting in lower stability of AR glasses operation.

[0092] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method of camera link determination for an AR glasses, the AR glasses comprising a plurality of cameras, the method comprising: The method includes: Multiple first relative pose relationships are obtained, each first relative pose relationship being the relative pose relationship between two adjacent cameras. The first relative pose relationship includes the position relationship and orientation vector relationship of the cameras at the current moment. Construct a spanning tree containing multiple of the aforementioned cameras; The weights of each edge of the spanning tree are calculated based on the first relative pose relationship, and the minimum spanning tree formed by the edge with the smallest weight is determined as the camera link at the current moment. Calculating the weights of each edge of the spanning tree based at least on the first relative pose relationship includes: calculating the distance between the projections of the optical centers of the two cameras corresponding to each edge of the spanning tree onto a predetermined plane, wherein the normal vector of the predetermined plane is the angle bisector of the orientation vectors of the two cameras; and determining the weight of the edge based on the distance, wherein the weight is proportional to the distance.

2. The method of claim 1, wherein, Calculating the weights of each edge of the generated tree based at least on the first relative pose relationship also includes: Determine whether there is an abnormal camera among the multiple cameras, wherein the abnormal camera is a faulty camera and / or an obstructed camera; If an abnormal camera exists among the two cameras corresponding to the edge, the weight of the edge corresponding to the abnormal camera is determined to be infinite. If there is no abnormal camera among the two cameras corresponding to the edge, calculate the distance between the projections of the optical centers of the two cameras corresponding to the edge onto the predetermined plane; The weight of the edge is determined based on the distance, and the weight is proportional to the distance.

3. The method of claim 2, wherein, Determining whether an abnormal camera exists among the multiple cameras includes: Acquire images from multiple cameras at the current moment to obtain multiple first images; The first image is divided into multiple image regions, and the grayscale value of each image region is calculated. Determine whether the difference in grayscale values ​​of the image regions corresponding to the same position in the first images of two adjacent cameras is within a first predetermined range; If the difference is within the first predetermined range, it is determined that the abnormal camera does not exist; If the difference is not within the first predetermined range, it is determined that there is an abnormal camera among the two adjacent cameras.

4. The method of claim 3, wherein, If it is determined that an abnormal camera exists among two adjacent cameras, determining the abnormal camera among the two adjacent cameras includes: Based on the two gray values ​​corresponding to the difference between two adjacent cameras that are not within the first predetermined range, an abnormal camera is determined among the two adjacent cameras. If one of the two gray values ​​is not within the second predetermined range, the camera in the image region corresponding to the gray value that is not within the second predetermined range is determined to be the abnormal camera.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The connection edges of the camera link at the previous moment and the camera link at the current moment are determined to obtain a camera link including the camera link at the previous moment, the connection edges, and the camera link at the current moment. The connection edge is the edge with the smallest weight connecting the camera link at the previous moment and the camera link at the current moment. The previous moment is the moment when the previous frame image was captured before the current moment.

6. The method according to claim 5, characterized in that, Determining the connection edges of the camera link at the previous moment and the camera link at the current moment includes: Obtain the second relative pose relationship of multiple cameras at the current time and the previous time, where each second relative pose relationship is the relative pose relationship of the camera at the current time and the previous time. The weights of a plurality of candidate edges are calculated based at least on the second relative pose relationship, wherein the candidate edge is any one of the edges connecting the camera link at the previous moment and the camera link at the current moment; The candidate edge with the smallest weight is determined as the connecting edge.

7. A camera link determination device for AR glasses, the AR glasses comprising multiple cameras, characterized in that, The device includes: The acquisition unit is used to acquire multiple first relative pose relationships, each of which is the relative pose relationship between two adjacent cameras. The first relative pose relationship includes the position relationship and orientation vector relationship of the cameras at the current time. A building unit is used to construct a spanning tree containing multiple of the cameras; The first determining unit is configured to calculate the weights of each edge of the spanning tree based at least on the first relative pose relationship, and determine the minimum spanning tree formed by the edge with the smallest weight as the camera link at the current moment. The first determining unit includes a first calculation module and a first determining module, wherein the first calculation module is used to calculate the distance between the projections of the optical centers of the two cameras corresponding to each edge of the spanning tree onto a predetermined plane, and the normal vector of the predetermined plane is the angle bisector of the orientation vectors of the two cameras; the first determining module is used to determine the weight of the edge based on the distance, and the weight is proportional to the distance.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed by a processor, it controls the device on which the computer storage medium is located to perform the camera link determination method for AR glasses according to any one of claims 1 to 6.

9. A processor, characterized in that, The processor is used to run a computer program, wherein the computer program executes the camera link determination method for AR glasses according to any one of claims 1 to 6.

10. An electronic device, characterized in that, A camera link determination device for AR glasses is provided, wherein the camera link determination device for AR glasses is used to perform the camera link determination method for AR glasses as described in any one of claims 1 to 6.

Images