Attitude information determination method, apparatus and device, and readable storage medium

By dividing the sampled images into reference images and reference images sets and constructing corresponding constraint information, the problem of low efficiency in geometric matching and constraint information processing between multiple sampled images in the prior art is solved, and efficient posture information is achieved Sure.

CN119027497APending Publication Date: 2024-11-26BEIJING SANKUAI ONLINE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310610203.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-26
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The prior art is difficult to efficiently determine the posture information corresponding to the image, especially in geometrical matching and constraint information processing between multiple sampled images, resulting in low computing efficiency.

Method used

By dividing multiple sampled images into reference images and reference images sets, the constraint information between the reference images and reference images is constructed based on each of the same geometric figures, the number of constraint information is reduced, and the determination efficiency of attitude information is improved.

Benefits of technology

It realizes efficient determination of the pose information of the sampled image, reduces the computational complexity, improves the processing speed, and significantly speeds up the determination process of the pose information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027497A_ABST
    Figure CN119027497A_ABST
Patent Text Reader

Abstract

The invention discloses an attitude information determination method, apparatus and device, and a readable storage medium, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a plurality of sampling images, wherein any sampling image comprises at least one geometric figure; the plurality of sampling images are divided to obtain a reference image and a reference image set, the reference image set comprises at least one reference image, and at least one same geometric figure exists between geometric figures included in the reference image and geometric figures included in any reference image; for the reference image and any reference image, constructing constraint information between the reference image and any reference image based on the same geometric figures; and determining attitude information of each sampling image based on constraint information between the reference image and each reference image. The constraint information of the reference image and the reference image is determined instead of the constraint information of the same geometric figure, the number of the constraint information is reduced, and the determination efficiency of the attitude information is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a method, device, equipment and readable storage medium for determining posture information. Background Art

[0002] Image is a common information carrier that can be used to carry information such as the shape, color, texture, and environment of an objective object. Generally, an image of an objective object can be captured by an image acquisition device such as a video camera, a camera, a mobile phone, or a scanner, and the posture information can be determined from the image. The posture information is used to describe the transformation relationship between the camera coordinate system and the world coordinate system. A virtual model of the objective object can be reconstructed using the posture information and the image of the objective object. Based on this, how to determine the posture information corresponding to the image has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The present application provides a method, device, equipment and readable storage medium for determining posture information, which can be used to determine the posture information corresponding to an image. The technical solution includes the following contents.

[0004] On the one hand, a method for determining posture information is provided, the method comprising:

[0005] Acquire a plurality of sample images, any of which includes at least one geometric figure;

[0006] Dividing the plurality of sampled images to obtain reference images and a benchmark image set, wherein the benchmark image set includes at least one benchmark image, and there is at least one identical geometric figure between the geometric figures included in the reference image and the geometric figures included in any benchmark image;

[0007] For the reference image and any one of the benchmark images, construct constraint information between the reference image and any one of the benchmark images based on respective identical geometric figures;

[0008] Based on the constraint information between the reference image and each benchmark image, the pose information of each sampled image is determined.

[0009] On the other hand, a device for determining posture information is provided, the device comprising:

[0010] An acquisition module, used for acquiring a plurality of sampled images, any of which includes at least one geometric figure;

[0011] A division module, used for dividing the plurality of sampled images to obtain a reference image and a benchmark image set, wherein the benchmark image set includes at least one benchmark image, and there is at least one identical geometric figure between the geometric figures included in the reference image and the geometric figures included in any benchmark image;

[0012] A construction module, configured to construct constraint information between the reference image and any one of the benchmark images based on respective identical geometric figures;

[0013] The determination module is used to determine the posture information of each sampled image based on the constraint information between the reference image and each benchmark image.

[0014] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above-mentioned methods for determining posture information.

[0015] On the other hand, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned methods for determining posture information.

[0016] On the other hand, a computer program or a computer program product is also provided, wherein at least one computer program is stored in the computer program or the computer program product, and the at least one computer program is loaded and executed by a processor to enable the electronic device to implement any of the above-mentioned methods for determining posture information.

[0017] The technical solution provided by this application brings at least the following beneficial effects:

[0018] In the technical solution provided by the present application, multiple sampled images are divided into a reference image and a benchmark image set, and at least one identical geometric figure exists between the reference image and any benchmark image in the benchmark image set. Based on each identical geometric figure, constraint information between the reference image and the benchmark image is constructed, so that the posture information of each sampled image is determined based on the constraint information between the reference image and each benchmark image. Since the constraint information between the reference image and the benchmark image is determined instead of the constraint information between the identical geometric figures, the amount of constraint information is reduced and the efficiency of determining the posture information is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 It is a schematic diagram of an implementation environment of a method for determining posture information provided in an embodiment of the present application;

[0021] Figure 2 is a flow chart of a method for determining posture information provided by an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of image acquisition provided by an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of the relationship between a reference image and a benchmark image provided in an embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of information of a common point provided in an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of information of a same line provided in an embodiment of the present application;

[0026] Figure 7 is a schematic diagram of a reference plane, a reference image, a reference image and constraint information provided by an embodiment of the present application;

[0027] Figure 8 is a relationship diagram between the root mean square of absolute attitude error and time of a different solution provided in an embodiment of the present application;

[0028] Fig. 9 is a schematic diagram of a real data set provided in an embodiment of the present application;

[0029] Fig.10 is a structural schematic diagram of a device for determining posture information provided in an embodiment of the present application;

[0030] Fig.11 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0031] Fig.12 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0033] Figure 1 is a schematic diagram of an implementation environment of a method for determining posture information provided in an embodiment of the present application, such as Figure 1 As shown, the implementation environment includes a terminal device 101 and a server 102. The method for determining posture information in the embodiment of the present application can be executed by the terminal device 101, or by the server 102, or by the terminal device 101 and the server 102 together.

[0034] The terminal device 101 can be a smart phone, a game console, a desktop computer, a tablet computer, a laptop computer, a smart TV, a smart car device, an intelligent voice interaction device, a smart home appliance, an aircraft airborne device, etc. The server 102 can be a single server, or a server cluster consisting of multiple servers, or any one of a cloud computing platform and a virtualization center, which is not limited in the embodiments of the present application. The server 102 can be connected to the terminal device 101 through a wired network or a wireless network. The server 102 can have functions such as data processing, data storage, and data transmission and reception, which are not limited in the embodiments of the present application. The number of terminal devices 101 and servers 102 is not limited and can be one or more.

[0035] The embodiment of the present application provides a method for determining posture information, which can be applied to the above implementation environment. Figure 2 Taking the flowchart of a method for determining posture information provided by an embodiment of the present application as an example, for ease of description, the terminal device 101 or the server 102 that executes the method for determining posture information in the embodiment of the present application is referred to as an electronic device, and the method can be executed by an electronic device. Figure 2 As shown, the method includes the following steps.

[0036] Step 201, obtaining a plurality of sampling images, any of which includes at least one geometric figure.

[0037] In the embodiment of the present application, an image acquisition device can be used to acquire images of any actual object to obtain multiple sampled images, and the contents of any two sampled images are the same or different. The embodiment of the present application does not limit the image acquisition device. Exemplarily, the image acquisition device is at least one of a light-sensing camera, an infrared camera, a depth camera, etc.

[0038] See also Figure 3 , Figure 3 is a schematic diagram of an image acquisition provided by an embodiment of the present application. Figure 3(1) shows that the image acquisition device acquires images of the building and the ground along the motion trajectory, and the pitch angle and roll angle of the image acquisition device show periodic changes. In a possible implementation, the image acquisition device is arranged on an aircraft, and the pitch angle and roll angle of the image acquisition device are the pitch angle and roll angle of the aircraft carrying the image acquisition device, and the aircraft can be a manned aircraft or a drone. Figure 3 (2) in FIG. 1 shows that the image acquisition device first rises vertically from the ground to a certain height (for example, 100 meters), and then captures images of the ground along the motion trajectory. Optionally, the resolution of the image acquisition device is 1280×800 pixels, and the focal length is 930 pixels. Figure 3 In (1), a total of 200 sample images were collected. Figure 3 In (2) of , a total of 400 sample images were collected.

[0039] Any sampling image includes at least one geometric figure, and any geometric figure is a coplanar point or a coplanar line. Among them, if a point on the real object is collected in at least two sampling images, the points corresponding to the point on the real object in these sampling images are coplanar. In other words, the coplanar point is the point corresponding to the same point on the real object in at least two sampling images. Similarly, if a line on the real object is collected in at least two sampling images, the lines corresponding to the line on the real object in these sampling images are coplanar. In other words, the coplanar line is the line corresponding to the same line on the real object in at least two sampling images. For the convenience of distinction, the point on the real object is called a real point, the line on the real object is called a real line, the point in the sampling image is called a sampling point, and the line in the sampling image is called a sampling line.

[0040] Optionally, the ground truth posture is obtained by an inertial navigation system (INS) with a real-time kinematic (RTK) global positioning system (GPS). For any sampled image, the sampled image is parsed based on the ground truth posture by three-dimensional reconstruction software to obtain the sampling points and / or sampling lines included in the sampled image. In this way, the sampling points included in each sampled image can be obtained, and these sampling points correspond to the camera coordinate system, and a sampling point in the camera coordinate system corresponds to an actual point in the world coordinate system. Each actual point in the world coordinate system can be called a point cloud. The point cloud is segmented by a point cloud library (PCL) to obtain at least one reference plane, any reference plane is the surface of an actual object (for example, the reference plane is the ground, the wall, etc.), and the reference plane includes at least one actual point. The coplanar points included in each sampled image are determined by the sampling points included in each sampled image and the actual points included in each reference plane. The coplanar lines included in each sampling image are determined by detecting the distances between the endpoints of the sampling lines included in each sampling image and each reference plane.

[0041] Step 202, divide the multiple sampled images to obtain reference images and a benchmark image set, the benchmark image set includes at least one benchmark image, and there is at least one identical geometric figure between the geometric figures included in the reference image and the geometric figures included in any benchmark image.

[0042] In the embodiment of the present application, the sampled images can be divided to obtain at least one reference image and a benchmark image set corresponding to each reference image, and the benchmark image set includes at least one benchmark image. It can be understood that any sampled image can be a reference image and / or a benchmark image. In other words, the reference image and the corresponding benchmark images are different images, and one reference image can be a benchmark image corresponding to another reference image.

[0043] As mentioned above, the sampled image includes at least one geometric figure. Since the reference image and the benchmark image are both sampled images, the reference image includes at least one geometric figure, and similarly, the benchmark image also includes at least one geometric figure. If a geometric figure is captured in both the reference image and the corresponding benchmark image, the geometric figure is the same geometric figure between the reference image and the corresponding benchmark image. Among them, there is at least one same geometric figure between the reference image and the corresponding benchmark image. Since any geometric figure is a coplanar point or a coplanar line, the reference image includes at least one coplanar point and / or at least one coplanar line, and the benchmark image also includes at least one coplanar point and / or at least one coplanar line. There is at least one same coplanar point (hereinafter referred to as the same point) between the reference image and the corresponding benchmark image, and / or there is at least one same coplanar line (hereinafter referred to as the same line) between the reference image and the corresponding benchmark image.

[0044] For example, reference image A corresponds to reference image set B, and reference image set B includes reference images B1 to B3, wherein reference image A includes coplanar points 1 to 3 and coplanar lines 1 to 4. If reference image B1 includes coplanar points 1, 5, and coplanar line 2, the same geometric figures between reference image A and reference image B1 are: coplanar point 1 and coplanar line 2; if reference image B2 includes coplanar points 4, 5, and coplanar lines 2 to 4, the same geometric figures between reference image A and reference image B2 are: coplanar lines 2 to 4.

[0045] The embodiment of the present application does not limit the division method. In an exemplary embodiment, each sampled image can be used as each reference image. For any reference image, a reference image set corresponding to the reference image is determined from each sampled image. Alternatively, the division is performed according to steps 2021 to 2023 (not shown in the figure) as shown below, that is, step 202 includes steps 2021 to 2023.

[0046] Step 2021, obtaining a first geometric graphics set, where the first geometric graphics set is an initial geometric graphics set or is obtained by updating the initial geometric graphics set, and the initial geometric graphics set includes the geometric graphics included in each sampled image.

[0047] In the embodiment of the present application, the initial geometric graphics set includes all the geometric graphics included in the sampled images, and the initial geometric graphics set can be denoted as Q. The first geometric graphics set is the initial geometric graphics set Q, or the initial geometric graphics set Q is updated according to the implementation principle similar to step 2024 below, and the updating method is not described here.

[0048] Step 2022: If the first geometric figure set satisfies the set conditions, at least one first image is determined from a plurality of sampled images, and the number of geometric figures in the first geometric figure set included in any first image is not less than a first quantity threshold.

[0049] The embodiment of the present application does not limit the first geometric figure set to satisfy the set condition. Exemplarily, the first geometric figure set satisfying the set condition means that: the number of geometric figures in the first geometric figure set is not 0, or the number of geometric figures in the first geometric figure set is greater than the set number.

[0050] If the first geometric figure set meets the set condition, then for any sampled image, the number of geometric figures in the first geometric figure set included in the sampled image is counted. If the number of geometric figures in the first geometric figure set included in the sampled image is not less than the first number threshold, the sampled image is determined to be the first image.

[0051] The embodiment of the present application does not limit the method for determining the first quantity threshold. Exemplarily, the first quantity threshold is a randomly generated value, or a value set according to manual experience, or the number of the first geometric figures in the first geometric figure set included in each sampled image after sorting the number of geometric figures. For example, the number of geometric figures in the first geometric figure set included in each sampled image is sorted from large to small, and the first number (i.e., the largest number) after sorting is used as the first quantity threshold. In this case, the first image is the sampled image with the largest number of geometric figures in the first geometric figure set among the sampled images.

[0052] Step 2023: for any first image, use any first image as a reference image, and determine a reference image set corresponding to the reference image from a plurality of sampled images.

[0053] In the embodiment of the present application, a first image is used as a reference image. For any sampled image remaining in the plurality of sampled images except any reference image, if the sampled image and the reference image include the same geometric figure, the sampled image is the reference image corresponding to the reference image. In this way, each reference image corresponding to the reference image can be determined, thereby obtaining a reference image set corresponding to the reference image. It can be understood that the reference image and the corresponding reference image include at least one same geometric figure.

[0054] Each reference image and the corresponding benchmark image set can be obtained by steps 2021 to 2023. Optionally, step 2023 further includes steps 2024 to 2026 (not shown in the figure).

[0055] Step 2024, deleting the geometric figures included in each first image from the first geometric figure set to obtain a second geometric figure set.

[0056] In the embodiment of the present application, for any geometric figure in the first geometric figure set, if the geometric figure is a geometric figure included in the first image, the geometric figure is deleted from the first geometric figure set, thereby deleting the geometric figures included in each first image from the first geometric figure set, and directly updating the first geometric figure set to obtain the second geometric figure set. If the first geometric figure set is an initial geometric figure set, updating the first geometric figure set is to update the initial geometric figure set; if the first geometric figure set is obtained by updating the initial geometric figure set, updating the first geometric figure set is to indirectly update the initial geometric figure set.

[0057] Step 2025: If the second geometric graphics set meets the set conditions, at least one second image is determined from multiple sampled images, and the number of geometric graphics in the second geometric graphics set included in any second image is greater than a second quantity threshold.

[0058] The embodiment of the present application does not limit the second geometric figure set to satisfy the set condition. Exemplarily, the second geometric figure set satisfying the set condition means that the number of geometric figures in the second geometric figure set is not 0, or the number of geometric figures in the second geometric figure set is greater than the set number.

[0059] If the second geometric figure set meets the set condition, then for any sampled image, the number of geometric figures in the second geometric figure set included in the sampled image is counted. If the number of geometric figures in the second geometric figure set included in the sampled image is not less than the second number threshold, the sampled image is determined to be the second image.

[0060] The embodiment of the present application does not limit the method for determining the second quantity threshold. For example, the second quantity threshold is a randomly generated value, or a value set according to manual experience, or the number of the first in a set order after sorting the number of geometric figures in the second geometric figure set included in each sampled image. For example, the number of geometric figures in the second geometric figure set included in each sampled image is sorted from large to small, and the first number (i.e., the largest number) after sorting is used as the second quantity threshold. In this case, the second image is the sampled image that includes the largest number of geometric figures in the second geometric figure set among the sampled images.

[0061] It should be noted that the implementation principle of step 2025 is similar to the implementation principle of step 2022, and can be seen in the description of step 2022, which will not be repeated here.

[0062] Step 2026: For any second image, use any second image as a reference image, and determine a reference image set corresponding to the reference image from a plurality of sampled images.

[0063] In the embodiment of the present application, a second image is used as a reference image to determine the reference image corresponding to the reference image. The implementation principle of step 2026 is similar to the implementation principle of step 2023, which can be seen in the description of step 2023 and will not be repeated here.

[0064] Among them, the above steps 2021 to 2026 can be summarized as the algorithm shown below.

[0065] Algorithm: Get set A and set T = {B i |i∈A}, where set A includes each reference image, set T includes the reference image set corresponding to each reference image, and B i is the reference image set corresponding to the i-th reference image;

[0066] / / Through the empty set Update to get set A and set T

[0067] Q←{geometric figures included in all sampled images}; / / Get the set Q by updating the initial geometric figure set, which includes the geometric figures included in all sampled images

[0068] when When executing

[0069] x i ← The pose information of the sampled image (i.e., the first image i) that contains the largest number of geometric figures in Q;

[0070] Q i ←x i The geometry captured by the corresponding sampled image (i.e., the first image i);

[0071] B i ← Include Q i A sampling image of any geometric figure in (i.e., a reference image);

[0072] A←A∪{i},T←T∪{B i}; / / Update A through A and i, and update B through T and i Update to get T

[0073] Q←QQ i ; / / Delete Q from Q i , get Q

[0074] Finish

[0075] Step 203: for the reference image and any benchmark image, based on each identical geometric figure, construct constraint information between the reference image and any benchmark image.

[0076] As mentioned above, the reference image and any of the benchmark images include at least one identical geometric figure. In the related art, constraint information of the identical geometric figure can be constructed based on information of any identical geometric figure in the reference image and information of any identical geometric figure in the benchmark image. In this way, the constraint information of each identical geometric figure is constructed, resulting in a large volume of constraint information and low efficiency in processing the constraint information.

[0077] In an embodiment of the present application, constraint information between the reference image and the benchmark image is constructed based on information of each identical geometric figure in the reference image and information of each identical geometric figure in the benchmark image, thereby reducing the volume of the constraint information and improving computational efficiency.

[0078] See also Figure 4 , Figure 4 is a schematic diagram of the relationship between a reference image and a benchmark image provided by an embodiment of the present application. Figure 4 In the figure, the reference image and the benchmark image 1 include three similar points, and the reference image and the benchmark image 2 also include three similar points.

[0079] In the embodiments of the present application, Figure 4 As shown in (1), based on the information of the three identical points in the reference image (corresponding to the three thick solid lines) and the information of the three identical points in the reference image 1 (corresponding to the three thin solid lines), the constraint information between the reference image and the reference image 1 is determined. Based on the information of the three identical points in the reference image (corresponding to the three thick solid lines) and the information of the three identical points in the reference image 2 (corresponding to the three thin dotted lines), the constraint information between the reference image and the reference image 1 is determined. Therefore, the embodiment of the present application can determine two constraint information.

[0080] In related technologies, such as Figure 4 As shown in (2), one constraint information is determined based on the information of an identical point in the reference image (corresponding to a thick solid line) and the information of the identical point in the reference image 1 (corresponding to a thin solid line). The second constraint information is determined based on the information of another identical point in the reference image (corresponding to a thin solid line) and the information of the identical point in the reference image 1 (corresponding to a thick solid line). Based on the same implementation principle, six constraint information can be determined.

[0081] It can be seen from this that the related art has a large amount of constraint information, resulting in low computing efficiency, while the embodiment of the present application has a small amount of constraint information, which can improve computing efficiency.

[0082] In a possible implementation, step 203 includes steps 2031 to 2033 (not shown in the figure).

[0083] Step 2031, for any identical geometric figure, obtain the reference weight corresponding to any identical geometric figure.

[0084] In the embodiment of the present application, the reference weight corresponding to any identical geometric figure in the reference image and the information of any identical geometric figure in the base image is determined. Wherein, an identical geometric figure can be an identical point or an identical line, and the information of the identical point and the information of the identical line are different, resulting in different reference weights corresponding to the identical point and the reference weights corresponding to the identical line. The reference weights corresponding to the identical point and the reference weights corresponding to the identical line are respectively introduced below.

[0085] In implementation A, any identical geometric figure is an identical point. In this case, step 2031 includes steps A1 and A2 (not shown in the figure).

[0086] Step A1, obtain a first vector and a second vector, the first vector is the direction vector of the line between the origin of the camera coordinate system where the reference image is located and the same point, and the second vector is the direction vector of the line between the origin of the camera coordinate system where any benchmark image is located and the same point.

[0087] The information of any identical point refers to the information of the reverse projection of the identical point on the image where it is located, and is also the direction vector of the line connecting the origin of the camera coordinate system corresponding to the image where the identical point is located and the identical point (i.e. the reverse projection mentioned above). Figure 5 , Figure 5 is a schematic diagram of information of a common point provided by an embodiment of the present application. Figure 5 As shown in (1), the origin is the origin of the camera coordinate system corresponding to the image where the same point is located, the line from the origin to the same point is the reverse projection of the same point on its image, and the direction vector is the information of the reverse projection.

[0088] Optionally, the information of the same point can be expressed as: Among them, K is the intrinsic matrix of the camera corresponding to the image at the same point. are the homogeneous coordinates of the same point p, that is

[0089] In the embodiment of the present application, the same point is located in the reference image and the reference image corresponding to the reference image. Therefore, the same point corresponds to two direction vectors, which are respectively called the first vector and the second vector. Figure 5As shown in (2), the first vector is the direction vector 1 of the line between the origin 1 of the camera coordinate system where the reference image is located and the same point, and the second vector is the direction vector 2 of the line between the origin 2 of the camera coordinate system where the reference image is located and the same point.

[0090] Alternatively, the first vector can be expressed as x1 is the horizontal coordinate of the first vector (i.e., the x coordinate), y1 is the vertical coordinate of the first vector (i.e., the y coordinate), and z1 is the vertical coordinate of the first vector (i.e., the z coordinate). Similarly, the second vector can be expressed as x2 is the horizontal coordinate of the second vector (ie, the x coordinate), y2 is the vertical coordinate of the second vector (ie, the y coordinate), and z2 is the vertical coordinate of the second vector (ie, the z coordinate).

[0091] Step A2: determining reference weights corresponding to the same points based on the first vector and the second vector.

[0092] In the embodiment of the present application, the following formula (1) can be used to calculate the first vector and the second vector Determine the reference weight c corresponding to the same point.

[0093] c=[x1x2,x1y2,x1z2,y1x2,y1y2,y1z2,z1x2,z1y2,z1z2] T Formula (1)

[0094] In implementation B, any identical geometric figures are identical lines. In this case, step 2031 includes steps B1 to B4 (not shown in the figure).

[0095] Step B1, obtain the third vector, the fourth vector and the fifth vector, the third vector is the normal vector of the plane formed by the origin of the camera coordinate system where the reference image is located and the same line, the fourth vector is the direction vector of the line between the origin of the camera coordinate system where any benchmark image is located and a vertex in the same line, and the fifth vector is the direction vector of the line between the origin of the camera coordinate system where any benchmark image is located and another vertex in the same line.

[0096] The information of any identical line can include the information of the identical points at both ends of the identical line. The information of any identical line can also refer to the information of the reverse projection of the identical line on the image where it is located, which is also the normal vector of the plane formed by the origin of the camera coordinate system corresponding to the image where the identical line is located and the identical line (that is, the reverse projection of the identical line on the image where it is located). See Figure 6 , Figure 6 Schematic diagram of information of a same line provided by an embodiment of the present application. Figure 6As shown in (1), the origin is the origin of the camera coordinate system corresponding to the image where the same line is located. The plane formed by the origin and the same line is the reverse projection of the same line on its image, and the normal vector of the plane is the information of the reverse projection.

[0097] Alternatively, the information of the same line can be expressed as: Among them, K is the intrinsic matrix of the camera corresponding to the image where the same line is located. l represents the same line.

[0098] In the embodiment of the present application, the same line is located in the reference image and the reference image corresponding to the reference image. The same line corresponds to two direction vectors, respectively referred to as the fourth vector and the fifth vector. The same line also corresponds to a normal vector, referred to as the third vector. Figure 6 As shown in (2), the third vector is the normal vector of the plane formed by the origin 1 of the camera coordinate system where the reference image is located and the same line 1, the fourth vector is the direction vector 1 of the line between the origin 2 of the camera coordinate system where the reference image is located and a vertex in the same line 2, and the fifth vector is the direction vector 2 of the line between the origin 2 of the camera coordinate system where the reference image is located and another vertex in the same line 2.

[0099] Optionally, the third vector can be represented as n l1 =[a1; b1; c1], the fourth vector or the fifth vector can be expressed as a1 is the abscissa of the third vector (ie, the x-coordinate), b1 is the ordinate of the third vector (ie, the y-coordinate), and c1 is the vertical coordinate of the third vector (ie, the z-coordinate). is the horizontal coordinate (i.e. x coordinate) of the fourth vector or the fifth vector, is the ordinate (i.e., y coordinate) of the fourth vector or the fifth vector, is the vertical coordinate (ie, z coordinate) of the fourth vector or the fifth vector. i represents the ith vertex of the same line l2 in the reference image. l1 represents the same line in the reference image.

[0100] Step B2: determining a first weight based on the third vector and the fourth vector.

[0101] In the embodiment of the present application, the third vector n can be obtained according to the following formula (2): l1 and the fourth vector Determine the first weight d i .

[0102]

[0103] Step B3, determining a second weight based on the third vector and the fifth vector.

[0104] It should be noted that the implementation principle of step B3 is similar to that of step B2, and the description of step B2 can be referred to and will not be repeated here.

[0105] Step B4: determining a reference weight corresponding to the same line based on the first weight and the second weight.

[0106] Optionally, a vector is formed by the first weight and the second weight, and the vector is a reference weight, that is, d = [d 1 ;d 2 ]. Among them, d 1 is the first weight, d 2 is the second weight.

[0107] Step 2032, obtaining a reference mapping relationship, the reference mapping relationship is used to describe the relationship between the reference plane, the posture information of the reference image and the posture information of any reference image, the reference plane is the surface of the actual object where each identical geometric figure is located.

[0108] If each identical geometric figure includes multiple identical points, the reference mapping relationship includes a first mapping relationship corresponding to the identical points. Optionally, the first mapping relationship can be expressed as f i (τ, x1, x2), i = 1, 2, 3. Among them, the first mapping relationship includes f1, f2 and f3. f1(τ, x1, x2) represents the relationship between the reference plane μ, the posture information x1 of the reference image and the posture information x2 of the reference image on the horizontal axis (i.e., the x-axis). f2(τ, x1, x2) represents the relationship between the reference plane τ, the posture information x1 of the reference image and the posture information x2 of the reference image on the vertical axis (i.e., the y-axis). f3(μ, x1, x2) represents the relationship between the reference plane τ, the posture information x1 of the reference image and the posture information x2 of the reference image on the vertical axis (i.e., the z-axis).

[0109] If each identical geometric figure includes a plurality of identical lines, the reference mapping relationship includes a second mapping relationship corresponding to the identical lines. Optionally, the second mapping relationship may be expressed as g(τ, x1, x2). The second mapping relationship g(τ, x1, x2) represents the relationship between the reference plane τ, the posture information x1 of the reference image, and the posture information x2 of the reference image.

[0110] If each identical geometric figure includes a plurality of identical points and a plurality of identical lines, the reference mapping relationship includes a first mapping relationship and a second mapping relationship.

[0111] Step 2033: construct constraint information between the reference image and any benchmark image based on the reference mapping relationship and the reference weights corresponding to each identical geometric figure.

[0112] In the embodiment of the present application, step 2033 can be implemented according to any one of implementation methods D1 to D3.

[0113] In implementation D1, each identical geometric figure includes multiple identical points and multiple identical lines, and the reference mapping relationship includes a first mapping relationship corresponding to the identical points and a second mapping relationship corresponding to the identical lines. In this case, step 2033 includes steps C1 to C3 (not shown in the figure).

[0114] Step C1: constructing first constraint information between a reference image and any benchmark image based on a first mapping relationship and reference weights corresponding to each identical point.

[0115] In the embodiment of the present application, the reference weights corresponding to the respective identical points are combined into a reference weight vector corresponding to the identical points. The reference weight vector corresponding to the identical points is multiplied by the first mapping relationship to obtain the first constraint information between the reference image and any benchmark image.

[0116] Understandably, see Figure 5 . Assume that a common point P is a three-dimensional point on the plane π = [n; d] and its corresponding points in the two images, where n represents the plane normal vector, d represents the distance between the camera origin and the plane, and the two images are p1 and p2, respectively, where the posture information corresponding to p1 includes rotation information R1 and translation information t1, and the posture information corresponding to p2 includes rotation information R2 and translation information t2. The direction of the back projection line of the common point P in p1 can be expressed as The direction of the back projection line of the same point P in p2 can be expressed as Then there is constraint information: × represents the cross product. By τ = dn, we can equate π = [n; d] and define Then the above constraint information can be transformed into: c·f i (τ,x1,x2)=0,i=1,2,3. Among them, f i In (τ, x1, x2), τ represents the plane π, x1 represents the posture information corresponding to p1, and x2 represents the posture information corresponding to p2. Based on this principle, the following formula (3) can be obtained.

[0117] The derivation process of the above constraint information is as follows:

[0118] The back projection lines of the same point P in p1 and p2 are L1 and L2 respectively. In the world coordinate system, the directions of L1 and L2 are and L1 and L2 pass through the origin of the camera coordinate system. In the world coordinate system, L1 and L2 pass through the three-dimensional points t1 and t2 respectively. P is the intersection of L2 and plane π. At this time, we can have: n T P+d=0. In addition, α=-n T t2-d, set up If P is on L1, In this case, it can be concluded that:

[0119]

[0120] definition Then a×b=0 3×1 . Assume that the nth elements of a and b are a n , b n , expand a, b, we can get: and is the nth row of R1, c n is a vector related to π, t1, R1, and t2. In this case, we can conclude that:

[0121]

[0122] τ, x1, x2 are vectors related to π, (R1, t1), (R2, t2) respectively. It can be clearly concluded that About The coefficients of the quadratic form are functions of τ, x1, and x2. Therefore, substituting And expand it, we can conclude that the elements of a×b can be transformed into c·f i (τ,x1,x2)=0.

[0123] Step C2: constructing second constraint information between the reference image and any benchmark image based on the second mapping relationship and the reference weights corresponding to each identical line.

[0124] In the embodiment of the present application, the reference weights corresponding to the respective same lines are combined into a reference weight vector corresponding to the same line. The reference weight vector corresponding to the same line is multiplied by the second mapping relationship to obtain the second constraint information between the reference image and any benchmark image.

[0125] Understandably, see Figure 6. Assume that an identical line L is a three-dimensional line on the plane π = [n; d] and its corresponding points in the two images, where n represents the plane normal vector, d represents the distance between the camera origin and the plane, and the two images are l1 and l2, where the posture information corresponding to l1 includes rotation information R1 and translation information t1, and the posture information corresponding to l2 includes rotation information R2 and translation information t2. The normal of the same line L in the back-projection plane in l1 can be expressed as n l1 , the directions of the back projection lines of the two endpoints of the same line L in l2 are expressed as and Then there is constraint information: · represents the dot product. Among them, By τ = dn, we can equate π = [n; d], and define n l1 =[a1;b1;c1], Then the above constraint information can be transformed into: i g(τ,x1,x2)=0,i=1,2. In g(τ,x1,x2), τ represents the plane π, x1 represents the posture information corresponding to l1, and x2 represents the posture information corresponding to l2. Based on this principle, the following formula (3) can be obtained.

[0126] The derivation process of the above constraint information is as follows:

[0127] In the embodiment of the present application, the reverse projection plane of l1 is Back projection line of l2 The two endpoints of In the world coordinate system, Along the normal n l1 Passing through the 3D point t1, Along the normal n l2 Passes through the three-dimensional point t2. The intersection point with plane π is At this point, you can have: In addition, α = -n T t2-d, set up exist On, then In this case, it can be concluded that:

[0128]

[0129] definition Then we have u·v i = 0. Assume u, v i The nth elements of n , Expand u, v i , we can conclude that: and is the nth row of R1, d n is a vector related to π, t1, R1, and t2. In this case, we can expand u·v i =0, and converted to d i g(τ,x1,x2)=0.

[0130] Step C3, determining the constraint information between the reference image and any benchmark image based on the first constraint information and the second constraint information between the reference image and any benchmark image.

[0131] In the embodiment of the present application, based on the first constraint information and the second constraint information between the reference image and any benchmark image, the constraint information between the reference image and the benchmark image is constructed. Optionally, the constraint information is shown in the following formula (3).

[0132]

[0133] ε(τ,x1,x2) represents the constraint information between the reference image and the benchmark image, τ represents the benchmark plane, x1 represents the pose information of the reference image, and x2 represents the pose information of the benchmark image. C represents the reference weight vector corresponding to the same point, C = [c1,c2,…,c N ] T ,c1,c2,…,c N represents the reference weights corresponding to the 1st to Nth identical points. D represents the reference weight vector corresponding to the identical lines. Characterizes the reference weight corresponding to the Mth identical line, Characterizes the first weight corresponding to the Mth identical line, Characterize the second weight corresponding to the Mth identical line. Cf1(τ, x1, x2), Cf2(τ, x1, x2), Cf3(τ, x1, x2) are the first constraint information between the reference image and the benchmark image, and Dg(τ, x1, x2) is the second constraint information between the reference image and the benchmark image.

[0134] It can be understood that on the reference plane, N points and M lines are captured with posture information x1 and x2 to obtain the point set and Line Set in, Represents the nth identical point. For the nth identical point, the reference weight c corresponding to the nth identical point can be calculated. n . Represents the mth identical line. For the mth identical line, the reference weight corresponding to the mth identical line can be calculated. Based on this, C and D shown in formula (3) can be obtained. All residuals of N points and M lines are superimposed to obtain a (3N+2M)-dimensional vector, namely formula (3). Since formula (3) is the constraint information between the reference image and the benchmark image, it includes the constraint information of each identical point and / or identical line in the reference image and the benchmark image. Regardless of the number of identical points and / or identical lines, the computational complexity of formula (3) is O(1), which can significantly reduce the running time.

[0135] In implementation method D2, each identical geometric figure includes a plurality of identical points. In this case, the constraint information between the reference image and the benchmark image is the first constraint information between the reference image and the benchmark image.

[0136] In implementation method D3, each identical geometric figure includes a plurality of identical lines. In this case, the constraint information between the reference image and the benchmark image is the second constraint information between the reference image and the benchmark image.

[0137] Step 204: determining the posture information of each sampled image based on the constraint information between the reference image and each benchmark image.

[0138] The pose information of any sampled image is the transformation relationship between the camera coordinate system of the camera corresponding to the sampled image and the world coordinate system, and the transformation relationship T satisfies T∈SE(3). Where SE(3) represents the rigid body transformation motion. The rigid body transformation motion includes the rotation component R and the translation component t, and R∈SO(3) and t∈R 3 Among them, SO(3) represents rotational motion, R 3 represents an array including three real numbers. Optionally, the posture information x of the sampled image satisfies x=[ω; t], wherein ω represents the angular axis of the rotation component R.

[0139] The problem of solving the constraint information between the reference image and each benchmark image belongs to the nonlinear least squares (NLLS) problem. In the embodiment of the present application, by solving the constraint information between the reference image and each benchmark image, the posture information of each sampled image and the description information of each benchmark plane are obtained, so that the sum of square errors of the constraint information between the reference image and each benchmark image can be minimized through the posture information of each sampled image and the description information of each benchmark plane. If the constraint information between the reference image and each benchmark image is e(x), the NLLS problem can be described as That is, find the independent variable x that minimizes the sum of squared errors of e(x), which is the posture information of each sampled image. For ease of description, the independent variable x in e(x) is omitted, that is, e(x) is equivalent to e.

[0140] See also Figure 7 , Figure 7 is a schematic diagram of a reference plane, a reference image, a reference image, and constraint information provided by an embodiment of the present application. Figure 7 It can be seen that one reference image and one base image correspond to one constraint information, and the constraint information corresponds to one base plane. Through each constraint information, description information of each base plane, posture information of each reference image, and posture information of each base image can be determined.

[0141] The following describes a method for determining the posture information of the sampled image. In a possible implementation, step 204 includes steps E1 to E4 (not shown in the figure).

[0142] Step E1, determining first posture information of each sampled image based on constraint information between the reference image and each benchmark image.

[0143] In the embodiment of the present application, the constraint information between the reference image and any benchmark image includes a reference mapping relationship, and the reference mapping relationship is used to describe the relationship between the benchmark plane, the posture information of the reference image, and the posture information of the benchmark image. Therefore, one reference image and one benchmark image correspond to one benchmark plane, and since there is at least one reference image, each reference image corresponds to at least one benchmark image, and therefore, there is also at least one benchmark plane.

[0144] Based on the Levenberg-Marquardt (LM) algorithm, the constraint information between the reference image and each benchmark image can be solved to obtain the first posture information of each sampled image and the first description information of each benchmark plane, and the first description information of the benchmark plane is used to describe the benchmark plane.

[0145] Optionally, step E1 includes steps E11 to E14 (not shown in the figure).

[0146] Step E11, based on the constraint information between the reference image and each benchmark image, determine the first factor, the second factor and the third factor, the first factor is the transpose of the reference Jacobian matrix of the constraint information between the reference image and each benchmark image multiplied by the reference Jacobian matrix, the second factor is the transpose of the reference Jacobian matrix multiplied by the constraint information between the reference image and each benchmark image, and the third factor is the transpose of the constraint information between the reference image and each benchmark image multiplied by the constraint information between the reference image and each benchmark image.

[0147] In the embodiment of the present application, the constraint information between the reference image and each benchmark image is e, then the transpose of e is e T , the reference Jacobian matrix of e is J e, the transpose of the reference Jacobian matrix is On this basis, the first factor can be determined based on e Second Factor The third factor e T e. In which, e is grouped so that a reference image and the benchmark image corresponding to the reference image form a group, and the group is recorded as e i .but And e i The Jacobian matrix is ​​J i .

[0148] In the LM algorithm, J is usually calculated first. e , then calculate and To construct the formula (6) mentioned below, then calculate To update the calculation. The computational complexity of this process is O(N). This embodiment of the application does not require J e and e, but calculate and These three parameters are used to speed up the calculation. Specifically, the constraint information between images is grouped into constraint information between reference images and baseline images, and a large amount of calculation is shared between each group in each iteration, which speeds up the calculation process by eliminating redundant calculations.

[0149] In an exemplary embodiment, the constraint information between the reference image and any of the benchmark images is constructed by a reference mapping relationship and reference weights corresponding to each identical geometric figure; the first factor includes a first sub-factor between the reference image and each benchmark image, the second factor includes a second sub-factor between the reference image and each benchmark image, and the third factor includes a third sub-factor between the reference image and each benchmark image. Step E11 includes steps E111 to E113 (not shown in the figure).

[0150] Step E111, for the constraint information between the reference image and any benchmark image, based on the reference weights corresponding to each identical geometric figure, determine the first parameter, which is the transpose of the reference weights corresponding to each identical geometric figure multiplied by the reference weights corresponding to each identical geometric figure.

[0151] If each identical geometric figure is an identical point, then the reference weight vector corresponding to each identical point (i.e., C mentioned above) is constructed by using the reference weight corresponding to each identical point. The transpose of C is multiplied by C to obtain the first parameter. Optionally, the first parameter K is K=C T C.

[0152] If each identical geometric figure is an identical line, then the reference weight vector corresponding to each identical line (ie, D mentioned above) is constructed by using the reference weight corresponding to each identical line. The transpose of D is multiplied by D to obtain the first parameter. Optionally, the first parameter Q is Q=D T D.

[0153] If each identical geometric figure includes each identical point and each identical line, the first parameter includes the aforementioned K and the aforementioned Q.

[0154] Step E112, based on the reference mapping relationship, determine the second parameter, the third parameter and the fourth parameter, the second parameter is the partial derivative of the reference mapping relationship relative to the reference plane, the third parameter is the partial derivative of the reference mapping relationship relative to the posture information of the reference image, and the fourth parameter is the partial derivative of the reference mapping relationship relative to the posture information of any reference image.

[0155] If the identical geometric figures are identical points, the reference mapping relationship is the first mapping relationship (i.e., the f i (τ,x1,x2), which can be abbreviated as f i ). In this case, the second parameter U i satisfy The third parameter V i satisfy The fourth parameter W i satisfy

[0156] If the identical geometric figures are identical lines, the reference mapping relationship is the second mapping relationship (i.e., g(τ, x1, x2) mentioned above, which can be abbreviated as g). In this case, the second parameter X satisfies The third parameter Y satisfies The fourth parameter Z satisfies

[0157] If each identical geometric figure includes each identical point and each identical line, the second parameter includes the aforementioned U i and the aforementioned X, the third parameter includes the aforementioned V i and the aforementioned Y, the fourth parameter includes the aforementioned W i and the aforementioned Z.

[0158] In the embodiment of the present application, it can be assumed that an n-dimensional vector ψ=[ψ1, ψ2, …, ψ N ] of M-dimensional function v=[v1,v2,…,v M ], that is, each element in v is a function related to ψ. Then the derivative of v with respect to ψ is an M×N matrix, which can be expressed as: Among them, σ ijis the entry in the i-th row and j-th column of the matrix. The implementation principle of the second, third and fourth parameters is the same as Similar, no further description is given here.

[0159] Step E113: determining a first sub-factor, a second sub-factor and a third sub-factor between the reference image and any benchmark image based on the first parameter, the second parameter, the third parameter and the fourth parameter.

[0160] There is at least one identical geometric figure between the geometric figures included in the reference image and the geometric figures included in any of the reference images. If each identical geometric figure is an identical point, then based on the first parameter K, the second parameter U i 、The third parameter V i and the fourth parameter W i , determine the target parameters as shown in the following formula (4).

[0161]

[0162] Next, based on the above target parameters, The formula to determine the first sub-factor Based on the aforementioned first parameter K and second parameter U i 、The third parameter V i , the fourth parameter W i And the first mapping relationship f i ,according to The formula to determine the second sub-factor Wherein, T represents the transposed matrix of the matrix. Based on the aforementioned first mapping relationship f i and the first parameter K, according to The formula to determine the third sub-factor ε T ε.

[0163] There is at least one identical geometric figure between the geometric figures included in the reference image and the geometric figures included in any of the benchmark images. If the identical geometric figures are identical lines, based on the first parameter Q, the second parameter X, the third parameter Y and the fourth parameter Z, the target parameter shown in the following formula (5) is determined.

[0164]

[0165] Next, based on the above target parameters, The formula to determine the first sub-factor Based on the first parameter Q, the second parameter X, the third parameter Y, the fourth parameter Z and the second mapping relationship g, The formula to determine the second sub-factor Wherein, T represents the transposed matrix of the matrix. Based on the aforementioned second mapping relationship g and the first parameter Q, according to ε T ε=g T The formula of Qg determines the third sub-factor ε T ε.

[0166] There is at least one identical geometric figure between the geometric figures included in the reference image and the geometric figures included in any of the reference images. If each identical geometric figure includes each identical point and each identical line, then based on the first parameter K, the second parameter U i 、The third parameter V i and the fourth parameter W i , the first parameter Q, the second parameter X, the third parameter Y and the fourth parameter Z, determine the target parameters shown in the above formulas (4) and (5).

[0167] Next, based on the above target parameters, The formula to determine the first sub-factor Based on the aforementioned first parameters K and Q, the second parameter U i and X, the third parameter V i and Y, the fourth parameter W i and Z, the first mapping relationship f i And the second mapping relationship g, according to The formula to determine the second sub-factor Wherein, T represents the transposed matrix of the matrix. Based on the aforementioned first mapping relationship f i , the second mapping relationship g and the first parameters K, Q, according to The formula to determine the third sub-factor ε T ε. The matrix of the above formula is relatively small, so the calculation efficiency is very high, and the calculation complexity of the first parameter, the first sub-factor, the second sub-factor and the third sub-factor is O(1). In addition, the first parameter is reused in the iteration process and only needs to be calculated once.

[0168] The Jacobian matrix has the following form:

[0169]

[0170] Substituting ε and the above J ε Substitution ε T ε, we can get

[0171] By the above method, the first sub-factor, the second sub-factor and the third sub-factor between the reference image and any benchmark image can be determined. The first factor is obtained by combining the first sub-factors. Based on the same principle, the second factor is obtained by combining the second sub-factors, and the third factor is obtained by combining the third sub-factors.

[0172] Step E12, determining an update factor through the first factor, the second factor and the third factor, wherein the update factor is used to update the second posture information of the sampled image and the second description information of the reference image in step E13.

[0173] In the embodiment of the present application, the target weight λ can be determined by the third factor. Second Factor and the target weight λ, the update factor δ is determined according to the following formula (6). N is the identity matrix of size n.

[0174]

[0175] Step E13, obtaining second posture information of each sampled image, where the second posture information of the sampled image is the initial posture information of the sampled image or is updated based on the initial posture information of the sampled image, and the initial posture information of the sampled image is a set value.

[0176] In the embodiment of the present application, the initial posture information of any sampled image is a set value. At the first update, the initial posture information of the sampled image is used as the second posture information of the sampled image. At the second and any subsequent updates (recorded as the nth update, where n is a positive integer greater than 1), the second posture information of the sampled image obtained by updating the initial posture information of the sampled image n-1 times can be obtained.

[0177] In addition, the second description information of each reference plane can also be obtained, and the second description information of any reference plane is used to describe the reference plane. The second description information of the reference plane is the initial description information of the reference plane or is updated based on the initial description information of the reference plane. The initial description information of the reference plane is a set value and is used to describe the reference plane.

[0178] In the first update, the initial description information of the reference plane is used as the second description information of the reference plane. In the second and any subsequent updates (recorded as the nth update, where n is a positive integer greater than 1), the second description information of the reference plane obtained by updating the initial description information of the reference plane n-1 times can be obtained.

[0179] Step E14, determining the first posture information of each sampled image based on the second posture information of each sampled image and the update factor.

[0180] In the embodiment of the present application, the second posture information of each sampled image and the second description information of each reference plane can be recorded together as x n , x n Add the update factor δ to get x n+1 , that is, x n+1 =x n +δ. Among them, x n+1 Characterize the first posture information of each sampled image and the first description information of each reference plane.

[0181] Step E2, based on the first posture information of each sampled image, determining a first index, the first index being used to describe the inner product of the constraint information between the reference image and each benchmark image in the case of the first posture information of each sampled image.

[0182] In the embodiment of the present application, by determining the first posture information of each sampled image and the first description information of each reference plane, any reference mapping relationship used to describe the relationship between the reference plane, the posture information of the reference image, and the posture information of any reference image can be determined, thereby determining the constraint information between the reference image and the reference image. In this way, the constraint information between the reference image and each reference image can be determined, thereby determining the first indicator.

[0183] Optionally, the first indicator is: Among them, x i Represents the i-th reference image, x j Represents the jth reference image corresponding to the i-th reference image, τ k represents the kth reference plane. A represents a reference image set, including at least one reference image. B i The reference image set corresponding to the i-th reference image includes at least one reference image. i,j A reference plane set corresponding to the i-th reference image and the j-th reference image is characterized, and includes at least one reference plane.

[0184] Step E3, obtaining a second indicator, where the second indicator is used to describe the setting error.

[0185] The embodiment of the present application does not limit the setting error. For example, the setting error includes at least one of the error of the inertial measurement unit (IMU), the error of the global positioning system (GPS), the reprojection error of non-coplanar points, the reprojection error of non-coplanar lines, etc. Optionally, the second indicator is C other .

[0186] Step E4, when the first index and the second index meet the reference condition, determine the first posture information of each sampling image as the posture information of each sampling image.

[0187] In an embodiment of the present application, the first indicator and the second indicator can be added to obtain an addition result. If the addition result meets the reference condition, the first posture information of each sampled image is determined as the posture information of each sampled image, and the first description information of each reference plane is used as the description information of each reference plane. The embodiment of the present application does not limit whether the addition result meets the reference condition. Exemplarily, the addition result meets the reference condition means that: the addition result is less than a set value, or the number of updates corresponding to the addition result reaches a set number of times.

[0188] In a possible implementation, the method further includes steps E5 to E7 (not shown in the figure).

[0189] Step E5, when the first index and the second index do not meet the reference condition, determine the third posture information of each sampling image based on the first posture information of each sampling image, the reference image and the constraint information between each benchmark image.

[0190] In the embodiment of the present application, the first indicator and the second indicator are added to obtain an addition result. If the addition result does not meet the reference condition, the first posture information of each sampled image is updated based on the constraint information between the reference image and each reference image to obtain the third posture information of each sampled image, and the first description information of each reference plane is updated to obtain the third description information of each reference plane. Among them, the implementation method of step E5 is similar to the implementation principle of step E1, which can be seen in the description of step E1 and will not be repeated here.

[0191] Step E6, based on the third posture information of each sampled image, determine a third index, the third index is used to describe the inner product of the constraint information between the reference image and each benchmark image in the case of the third posture information of each sampled image. The implementation method of step E6 is similar to the implementation principle of step E2, which can be seen in the description of step E2, and will not be repeated here.

[0192] Step E7: when the third index and the second index meet the reference condition, the third posture information of each sampling image is determined as the posture information of each sampling image.

[0193] The third index and the second index may be added to obtain an addition result. If the addition result satisfies the reference condition, the third posture information of each sampled image is determined as the posture information of each sampled image, and the third description information of each reference plane is used as the description information of each reference plane.

[0194] In an embodiment of the present application, synthetic data and real data are used to perform calculations on a computer with a processor and 64 Gigabyte (GB) memory. The present application embodiment compares three schemes, Scheme 1 is a method provided in an embodiment of the present application, Scheme 2 is to construct constraint information between each identical point and constraint information between each identical line, calculate these constraint information, obtain the posture information of each sampled image and the description information of each reference plane, and Scheme 3 is a traditional bundle adjustment (BA) algorithm with reprojection error. These three schemes correspond to the same reference conditions to achieve the same iteration end conditions. In addition, the root mean square of absolute pose errors (RMSAPE) is used to evaluate the convergence speed of each scheme, that is, the similarity transformation matrix is ​​used to align the posture information and description information after each iteration, and the performance is evaluated by calculating RMSAPE.

[0195] In the embodiment of the present application, the initial posture information of each sampled image and the position information of the landmark (point or end point of a line) are generated by perturbing the real value of the reference plane (e.g., the ground, wall, etc.) with isotropic zero-mean Gaussian noise. The standard deviations (STD) of the Gaussian noise of the translation information, the angular axis of the rotation information, and the landmark are represented by σ t , σ θ and σ l ,in, The initial description information of the reference plane is obtained by plane fitting of points and lines.

[0196] First, we use synthetic data to compare the performance of different schemes. Figure 3 The method shown collects multiple sample images and adds zero-mean Gaussian noise with a STD of 1 pixel to the two-dimensional points and the two end points of the two-dimensional line. The initialization of the initial posture information and the location information of the landmark is completed by perturbation. For the three-dimensional points and three-dimensional lines, the STD of the Gaussian noise is set to σ l =0.1m. For the initial posture information, set σt =2m,σ θ =0.2rad, the result is as follows Figure 8 As shown in (1) in . Figure 8 It can be seen from (1) that, under the condition of the same absolute attitude error root mean square, solution (1) requires less time. Therefore, the method of the embodiment of the present application can speed up the calculation.

[0197] Next, real data is used to compare the performance of different schemes. Optionally, four real datasets are collected in an urban environment using two aerial robots, e.g. Fig. 9 As shown, Fig. 9 (1) to (4) in the figure contain 306, 377, 757, and 895 images, respectively. The ground truth pose is obtained by the inertial navigation system. Based on the ground truth pose, 3D points and 3D lines are generated, and planes are detected from the point cloud. By detecting the distance between the plane and the endpoints of the 3D line, coplanar lines are obtained. These coplanar points and coplanar lines are used to evaluate the performance of different schemes. The initial pose information and the location information of the landmarks are perturbed by the same Gaussian noise as the synthetic data, i.e., σ t =2m,σ θ =0.2rad,σ l =0.1m, the result is as follows Figure 8 As shown in (2) in . Figure 8 It can be seen from (2) that, under the condition of the same absolute attitude root mean square error, solution (1) requires less time. Therefore, the method of the embodiment of the present application can speed up the calculation.

[0198] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant regions. For example, the sample images involved in this application are all obtained with full authorization.

[0199] In the above method, multiple sampled images are divided into a reference image and a benchmark image set, and at least one identical geometric figure exists between the reference image and any benchmark image in the benchmark image set. Constraint information between the reference image and the benchmark image is constructed based on each identical geometric figure, so that the posture information of each sampled image is determined based on the constraint information between the reference image and each benchmark image. Since the constraint information between the reference image and the benchmark image is determined instead of the constraint information between the identical geometric figures, the amount of constraint information is reduced and the efficiency of determining the posture information is improved.

[0200] Fig.10FIG. 1 is a schematic diagram of a structure of a device for determining posture information provided by an embodiment of the present application. Fig.10 As shown, the device comprises:

[0201] An acquisition module 1001 is used to acquire a plurality of sample images, wherein any sample image includes at least one geometric figure;

[0202] A division module 1002 is used to divide the multiple sampled images to obtain reference images and a benchmark image set, wherein the benchmark image set includes at least one benchmark image, and there is at least one identical geometric figure between the geometric figures included in the reference image and the geometric figures included in any benchmark image;

[0203] A construction module 1003 is used to construct constraint information between the reference image and any benchmark image based on respective identical geometric figures;

[0204] The determination module 1004 is used to determine the posture information of each sampled image based on the constraint information between the reference image and each benchmark image.

[0205] In a possible implementation, the number of reference images is at least one, and any reference image corresponds to a benchmark image set;

[0206] The division module 1002 is used to obtain a first geometric figure set, where the first geometric figure set is an initial geometric figure set or is obtained by updating the initial geometric figure set, and the initial geometric figure set includes geometric figures included in each sampling image; if the first geometric figure set meets the set conditions, at least one first image is determined from multiple sampling images, and the number of geometric figures in the first geometric figure set included in any first image is not less than a first quantity threshold; for any first image, any first image is used as a reference image, and a reference image set corresponding to the reference image is determined from multiple sampling images.

[0207] In a possible implementation, the device further includes:

[0208] A deleting module, used for deleting the geometric figures included in each first image from the first geometric figure set to obtain a second geometric figure set;

[0209] The determination module 1004 is further configured to determine, if the second geometric figure set satisfies a set condition, at least one second image from the plurality of sampled images, wherein the number of geometric figures in the second geometric figure set included in any second image is greater than a second number threshold;

[0210] The determination module 1004 is further configured to take any second image as a reference image and determine a reference image set corresponding to the reference image from a plurality of sampled images.

[0211] In one possible implementation, construction module 1003 is used to obtain, for any identical geometric figure, a reference weight corresponding to any identical geometric figure; obtain a reference mapping relationship, the reference mapping relationship is used to describe the relationship between a reference plane, posture information of a reference image, and posture information of any reference image, the reference plane being the surface of an actual object where each identical geometric figure is located; based on the reference mapping relationship and the reference weights corresponding to each identical geometric figure, construct constraint information between the reference image and any reference image.

[0212] In a possible implementation, any identical geometric figure is an identical point;

[0213] Construction module 1003 is used to obtain a first vector and a second vector, where the first vector is a direction vector of a line between the origin of a camera coordinate system where the reference image is located and the same point, and the second vector is a direction vector of a line between the origin of a camera coordinate system where any reference image is located and the same point; based on the first vector and the second vector, a reference weight corresponding to the same point is determined.

[0214] In a possible implementation, any identical geometric figure is an identical line;

[0215] Construction module 1003 is used to obtain a third vector, a fourth vector and a fifth vector, wherein the third vector is a normal vector of a plane formed by the origin of the camera coordinate system where the reference image is located and the same line, the fourth vector is a direction vector of a line between the origin of the camera coordinate system where any reference image is located and a vertex in the same line, and the fifth vector is a direction vector of a line between the origin of the camera coordinate system where any reference image is located and another vertex in the same line; based on the third vector and the fourth vector, a first weight is determined; based on the third vector and the fifth vector, a second weight is determined; based on the first weight and the second weight, a reference weight corresponding to the same line is determined.

[0216] In a possible implementation, each identical geometric figure includes a plurality of identical points and a plurality of identical lines, and the reference mapping relationship includes a first mapping relationship corresponding to the identical points and a second mapping relationship corresponding to the identical lines;

[0217] Construction module 1003 is used to construct first constraint information between the reference image and any benchmark image based on the first mapping relationship and the reference weights corresponding to each identical point; construct second constraint information between the reference image and any benchmark image based on the second mapping relationship and the reference weights corresponding to each identical line; determine the constraint information between the reference image and any benchmark image based on the first constraint information and the second constraint information between the reference image and any benchmark image.

[0218] In one possible implementation, the determination module 1004 is used to determine the first posture information of each sampling image based on the constraint information between the reference image and each benchmark image; determine a first indicator based on the first posture information of each sampling image, the first indicator being used to describe the inner product of the constraint information between the reference image and each benchmark image in the case of the first posture information of each sampling image; obtain a second indicator, the second indicator being used to describe the setting error; and determine the first posture information of each sampling image as the posture information of each sampling image when the first indicator and the second indicator meet the reference condition.

[0219] In a possible implementation, the determination module 1004 is used to determine a first factor, a second factor, and a third factor based on the constraint information between the reference image and each benchmark image, wherein the first factor is the transpose of the reference Jacobian matrix of the constraint information between the reference image and each benchmark image multiplied by the reference Jacobian matrix, the second factor is the transpose of the reference Jacobian matrix multiplied by the constraint information between the reference image and each benchmark image, and the third factor is the transpose of the constraint information between the reference image and each benchmark image multiplied by the constraint information between the reference image and each benchmark image; an update factor is determined through the first factor, the second factor, and the third factor; second posture information of each sampled image is obtained, the second posture information of the sampled image is the initial posture information of the sampled image or is updated based on the initial posture information of the sampled image, and the initial posture information of the sampled image is a set value; first posture information of each sampled image is determined based on the second posture information of each sampled image and the update factor.

[0220] In a possible implementation, the constraint information between the reference image and any benchmark image is constructed by a reference mapping relationship and reference weights corresponding to each identical geometric figure; the first factor includes a first sub-factor between the reference image and each benchmark image, the second factor includes a second sub-factor between the reference image and each benchmark image, and the third factor includes a third sub-factor between the reference image and each benchmark image;

[0221] Determination module 1004 is used to determine the first parameter for the constraint information between the reference image and any benchmark image based on the reference weights corresponding to the identical geometric figures, the first parameter being the transpose of the reference weights corresponding to the identical geometric figures multiplied by the reference weights corresponding to the identical geometric figures; determine the second parameter, the third parameter and the fourth parameter based on the reference mapping relationship, the second parameter being the partial derivative of the reference mapping relationship with respect to the benchmark plane, the third parameter being the partial derivative of the reference mapping relationship with respect to the posture information of the reference image, and the fourth parameter being the partial derivative of the reference mapping relationship with respect to the posture information of any benchmark image; determine the first sub-factor, the second sub-factor and the third sub-factor based on the reference image and any benchmark image based on the first parameter, the second parameter, the third parameter and the fourth parameter.

[0222] In a possible implementation, the determination module 1004 is also used to determine the third posture information of each sampling image based on the first posture information of each sampling image, the constraint information between the reference image and each benchmark image when the first indicator and the second indicator do not meet the reference condition; determine the third indicator based on the third posture information of each sampling image, the third indicator being used to describe the inner product of the constraint information between the reference image and each benchmark image when the third posture information of each sampling image is present; and determine the third posture information of each sampling image as the posture information of each sampling image when the third indicator and the second indicator meet the reference condition.

[0223] In the above device, multiple sampled images are divided into a reference image and a benchmark image set, and at least one identical geometric figure exists between the reference image and any benchmark image in the benchmark image set. Constraint information between the reference image and the benchmark image is constructed based on each identical geometric figure, so that the posture information of each sampled image is determined based on the constraint information between the reference image and each benchmark image. Since the constraint information between the reference image and the benchmark image is determined instead of the constraint information between the identical geometric figures, the amount of constraint information is reduced and the efficiency of determining the posture information is improved.

[0224] It should be understood that the above Fig.10 When the device provided realizes its functions, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0225] Fig.11The structure block diagram of a terminal device 1100 provided by an exemplary embodiment of the present application is shown. The terminal device 1100 includes: a processor 1101 and a memory 1102 .

[0226] The processor 1101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1101 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1101 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0227] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1102 is used to store at least one computer program, which is executed by the processor 1101 to implement the method for determining the posture information provided in the method embodiment of the present application.

[0228] In some embodiments, the terminal device 1100 may further optionally include: a peripheral device interface 1103 and at least one peripheral device. The processor 1101, the memory 1102 and the peripheral device interface 1103 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1103 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107 and a power supply 1108.

[0229] The peripheral device interface 1103 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1101 and the memory 1102. In some embodiments, the processor 1101, the memory 1102, and the peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1101, the memory 1102, and the peripheral device interface 1103 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0230] The radio frequency circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1104 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1104 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1104 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1104 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1104 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0231] The display screen 1105 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1105 is a touch display screen, the display screen 1105 also has the ability to collect touch signals on the surface or above the surface of the display screen 1105. The touch signal can be input to the processor 1101 as a control signal for processing. At this time, the display screen 1105 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1105 can be one, which is set on the front panel of the terminal device 1100; in other embodiments, the display screen 1105 can be at least two, which are respectively set on different surfaces of the terminal device 1100 or are folded; in other embodiments, the display screen 1105 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal device 1100. Even, the display screen 1105 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1105 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0232] The camera assembly 1106 is used to capture images or videos. Optionally, the camera assembly 1106 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1106 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0233] The audio circuit 1107 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1101 for processing, or input them into the radio frequency circuit 1104 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal device 1100. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1101 or the radio frequency circuit 1104 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1107 may also include a headphone jack.

[0234] The power supply 1108 is used to power various components in the terminal device 1100. The power supply 1108 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1108 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0235] In some embodiments, the terminal device 1100 further includes one or more sensors 1109 , including but not limited to: an acceleration sensor 1111 , a gyroscope sensor 1112 , a pressure sensor 1113 , an optical sensor 1114 , and a proximity sensor 1115 .

[0236] The acceleration sensor 1111 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal device 1100. For example, the acceleration sensor 1111 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1101 can control the display screen 1105 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1111. The acceleration sensor 1111 can also be used to collect game or user motion data.

[0237] The gyroscope sensor 1112 can detect the body direction and rotation angle of the terminal device 1100, and the gyroscope sensor 1112 can cooperate with the acceleration sensor 1111 to collect the user's 3D actions on the terminal device 1100. The processor 1101 can implement the following functions based on the data collected by the gyroscope sensor 1112: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0238] The pressure sensor 1113 can be set on the side frame of the terminal device 1100 and / or the lower layer of the display screen 1105. When the pressure sensor 1113 is set on the side frame of the terminal device 1100, the user's holding signal of the terminal device 1100 can be detected, and the processor 1101 performs left and right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 1113. When the pressure sensor 1113 is set on the lower layer of the display screen 1105, the processor 1101 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1105. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0239] The optical sensor 1114 is used to collect the ambient light intensity. In one embodiment, the processor 1101 can control the display brightness of the display screen 1105 according to the ambient light intensity collected by the optical sensor 1114. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1105 is increased; when the ambient light intensity is low, the display brightness of the display screen 1105 is reduced. In another embodiment, the processor 1101 can also dynamically adjust the shooting parameters of the camera assembly 1106 according to the ambient light intensity collected by the optical sensor 1114.

[0240] The proximity sensor 1115, also known as a distance sensor, is usually arranged on the front panel of the terminal device 1100. The proximity sensor 1115 is used to collect the distance between the user and the front of the terminal device 1100. In one embodiment, when the proximity sensor 1115 detects that the distance between the user and the front of the terminal device 1100 is gradually decreasing, the processor 1101 controls the display screen 1105 to switch from the screen-on state to the screen-off state; when the proximity sensor 1115 detects that the distance between the user and the front of the terminal device 1100 is gradually increasing, the processor 1101 controls the display screen 1105 to switch from the screen-off state to the screen-on state.

[0241] Those skilled in the art will understand that Fig.11 The structure shown in the figure does not constitute a limitation on the terminal device 1100, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0242] Fig.12This is a schematic diagram of the structure of the server provided in the embodiment of the present application. The server 1200 may have relatively large differences due to different configurations or performances, and may include one or more processors 1201 and one or more memories 1202, wherein the one or more memories 1202 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1201 to implement the method for determining the posture information provided in the above-mentioned various method embodiments. Exemplarily, the processor 1201 is a CPU. Of course, the server 1200 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 1200 may also include other components for implementing device functions, which will not be described in detail here.

[0243] In an exemplary embodiment, a computer-readable storage medium is further provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned methods for determining posture information.

[0244] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0245] In an exemplary embodiment, a computer program or a computer program product is also provided, in which at least one computer program is stored, and the at least one computer program is loaded and executed by a processor to enable the electronic device to implement any of the above-mentioned methods for determining posture information.

[0246] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0247] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0248] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining posture information, characterized in that: The method comprises: Acquire a plurality of sample images, any of which includes at least one geometric figure; Dividing the plurality of sampled images to obtain reference images and a benchmark image set, wherein the benchmark image set includes at least one benchmark image, and there is at least one identical geometric figure between the geometric figures included in the reference image and the geometric figures included in any benchmark image; For the reference image and any one of the benchmark images, construct constraint information between the reference image and any one of the benchmark images based on respective identical geometric figures; Based on the constraint information between the reference image and each benchmark image, the pose information of each sampled image is determined.

2. The method according to claim 1, characterized in that: The number of the reference images is at least one, and any reference image corresponds to a benchmark image set; The step of dividing the plurality of sampled images to obtain a reference image and a benchmark image set comprises: Obtain a first geometric graphics set, where the first geometric graphics set is an initial geometric graphics set or is obtained by updating the initial geometric graphics set, and the initial geometric graphics set includes the geometric graphics included in each of the sampled images; If the first geometric graphics set meets the set condition, at least one first image is determined from the multiple sampled images, and the number of geometric graphics in the first geometric graphics set included in any first image is not less than a first quantity threshold; For any first image, the any first image is used as the reference image, and a reference image set corresponding to the reference image is determined from the multiple sample images.

3. The method according to claim 2, characterized in that The method further comprises: Deleting the geometric figures included in each first image from the first geometric figure set to obtain a second geometric figure set; If the second geometric graphics set satisfies the set condition, determining at least one second image from the plurality of sampled images, wherein the number of geometric graphics in the second geometric graphics set included in any second image is greater than a second quantity threshold; For any second image, the any second image is used as the reference image, and a reference image set corresponding to the reference image is determined from the multiple sample images.

4. The method according to claim 1, characterized in that: The step of constructing constraint information between the reference image and any one of the benchmark images based on the respective identical geometric figures includes: For any identical geometric figure, obtaining a reference weight corresponding to the identical geometric figure; Acquire a reference mapping relationship, where the reference mapping relationship is used to describe the relationship between a reference plane, the posture information of the reference image, and the posture information of any one of the reference images, where the reference plane is a surface of an actual object where each of the identical geometric figures is located; Based on the reference mapping relationship and the reference weights corresponding to the respective identical geometric figures, constraint information between the reference image and any of the benchmark images is constructed.

5. The method according to claim 4, characterized in that The any one of the same geometric figures is a same point; and the obtaining of the reference weight corresponding to the any one of the same geometric figures comprises: Obtain a first vector and a second vector, wherein the first vector is a direction vector of a line between the origin of the camera coordinate system where the reference image is located and the same point, and the second vector is a direction vector of a line between the origin of the camera coordinate system where any one of the reference images is located and the same point; Based on the first vector and the second vector, a reference weight corresponding to the same point is determined.

6. The method according to claim 4, characterized in that Any of the identical geometric figures is an identical line; The obtaining of the reference weight corresponding to any one of the same geometric figures includes: Obtain a third vector, a fourth vector, and a fifth vector, wherein the third vector is a normal vector of a plane formed by the origin of the camera coordinate system where the reference image is located and the same line, the fourth vector is a direction vector of a line between the origin of the camera coordinate system where any one of the reference images is located and a vertex in the same line, and the fifth vector is a direction vector of a line between the origin of the camera coordinate system where any one of the reference images is located and another vertex in the same line; Determining a first weight based on the third vector and the fourth vector; Determining a second weight based on the third vector and the fifth vector; Based on the first weight and the second weight, a reference weight corresponding to the same line is determined.

7. The method according to claim 4, characterized in that The respective identical geometric figures include a plurality of identical points and a plurality of identical lines, and the reference mapping relationships include a first mapping relationship corresponding to the identical points and a second mapping relationship corresponding to the identical lines; The step of constructing constraint information between the reference image and any one of the benchmark images based on the reference mapping relationship and the reference weights corresponding to the respective identical geometric figures includes: constructing first constraint information between the reference image and any one of the benchmark images based on the first mapping relationship and the reference weights corresponding to the respective identical points; constructing second constraint information between the reference image and any one of the benchmark images based on the second mapping relationship and the reference weights corresponding to the respective identical lines; Based on the first constraint information and the second constraint information between the reference image and any one of the benchmark images, the constraint information between the reference image and any one of the benchmark images is determined.

8. The method according to claim 1, characterized in that The step of determining the posture information of each sampled image based on the constraint information between the reference image and each benchmark image comprises: Determining first posture information of each sampled image based on constraint information between the reference image and each benchmark image; Based on the first posture information of each sampled image, determining a first index, the first index being used to describe the inner product of constraint information between the reference image and each benchmark image in the case of the first posture information of each sampled image; Obtaining a second indicator, where the second indicator is used to describe a setting error; When the first index and the second index satisfy a reference condition, the first posture information of each sampling image is determined as the posture information of each sampling image.

9. The method according to claim 8, characterized in that The step of determining first posture information of each sampled image based on constraint information between the reference image and each benchmark image comprises: Based on the constraint information between the reference image and each benchmark image, determining a first factor, a second factor and a third factor, wherein the first factor is the transpose of a reference Jacobian matrix of the constraint information between the reference image and each benchmark image multiplied by the reference Jacobian matrix, the second factor is the transpose of the reference Jacobian matrix multiplied by the constraint information between the reference image and each benchmark image, and the third factor is the transpose of the constraint information between the reference image and each benchmark image multiplied by the constraint information between the reference image and each benchmark image; Determine an update factor by using the first factor, the second factor and the third factor; Acquire second posture information of each sampling image, wherein the second posture information of the sampling image is initial posture information of the sampling image or is updated based on the initial posture information of the sampling image, and the initial posture information of the sampling image is a set value; Based on the second posture information of the respective sample images and the update factor, the first posture information of the respective sample images is determined.

10. The method according to claim 9, characterized in that The constraint information between the reference image and any of the benchmark images is constructed by a reference mapping relationship and reference weights corresponding to the respective identical geometric figures; the first factor includes a first sub-factor between the reference image and the respective benchmark images, the second factor includes a second sub-factor between the reference image and the respective benchmark images, and the third factor includes a third sub-factor between the reference image and the respective benchmark images; The determining of the first factor, the second factor and the third factor based on the constraint information between the reference image and each benchmark image comprises: For the constraint information between the reference image and any one of the benchmark images, based on the reference weights corresponding to the respective identical geometric figures, determine a first parameter, the first parameter being the transpose of the reference weights corresponding to the respective identical geometric figures multiplied by the reference weights corresponding to the respective identical geometric figures; Based on the reference mapping relationship, determine a second parameter, a third parameter and a fourth parameter, wherein the second parameter is a partial derivative of the reference mapping relationship relative to the reference plane, the third parameter is a partial derivative of the reference mapping relationship relative to the posture information of the reference image, and the fourth parameter is a partial derivative of the reference mapping relationship relative to the posture information of any one of the reference images; Based on the first parameter, the second parameter, the third parameter and the fourth parameter, a first sub-factor, a second sub-factor and a third sub-factor based on the difference between the reference image and any benchmark image are determined.

11. The method according to claim 8, characterized in that The method further comprises: In the case where the first index and the second index do not satisfy the reference condition, determining third posture information of each sampling image based on the first posture information of each sampling image and the constraint information between the reference image and each benchmark image; Based on the third posture information of each of the sampled images, determining a third index, wherein the third index is used to describe the inner product of the constraint information between the reference image and each of the benchmark images in the case of the third posture information of each of the sampled images; When the third index and the second index satisfy the reference condition, the third posture information of each sampling image is determined as the posture information of each sampling image.

12. A device for determining posture information, characterized in that: The device comprises: An acquisition module, used for acquiring a plurality of sampled images, any of which includes at least one geometric figure; A division module, used for dividing the plurality of sampled images to obtain a reference image and a benchmark image set, wherein the benchmark image set includes at least one benchmark image, and there is at least one identical geometric figure between the geometric figures included in the reference image and the geometric figures included in any benchmark image; A construction module, configured to construct constraint information between the reference image and any one of the benchmark images based on respective identical geometric figures; The determination module is used to determine the posture information of each sampled image based on the constraint information between the reference image and each benchmark image.

13. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method for determining posture information as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method for determining posture information as described in any one of claims 1 to 11.