Spacecraft pose tracking method and system based on monocular vision and three-dimensional geometrical characteristics

Abstract

The invention relates to the technical field of computer vision, and discloses a spacecraft pose tracking method and system based on monocular vision and three-dimensional geometric features. The method specifically comprises the following steps: extracting three-dimensional geometric features including three-dimensional edges and three-dimensional contours from a spacecraft three-dimensional model, and performing approximation of any precision on each three-dimensional geometric feature by using a trigonometric polynomial to obtain a corresponding analytic parameter equation; and sampling the control points based on the analytic parameter equation, and obtaining a maximum likelihood estimation value of the spacecraft pose in a manner of minimizing a ghosting error between the planar projection of the control points and the corresponding image edge points. And correcting the maximum likelihood estimation value of the spacecraft pose by using an extended Kalman filtering model based on second-order autoregression to obtain a final estimation value of the spacecraft pose. According to the method, high-precision and high-efficiency spacecraft tracking can be realized by virtue of very low cost, and the method has very strong robustness for shadow shielding, different backgrounds, distance changes and image noise.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and system for tracking a spacecraft posture based on monocular vision and three-dimensional geometric features. Background Art

[0002] With the advancement of space technology, the demand for on-orbit spacecraft services is increasing, including inspection, maintenance, and on-orbit servicing. The core of these tasks is the ability for the service spacecraft to safely and accurately approach the target spacecraft and measure the relative six-degree-of-freedom pose (position and attitude) of the two in real time. Monocular vision-based pose estimation methods, due to their advantages such as real-time performance, high accuracy, and low cost, have become the optimal solution for approach navigation.

[0003] Existing monocular vision-based pose tracking algorithms are mainly divided into four categories: keypoint-based methods, regression-based methods, region-based methods, and edge-based methods. Among them, edge-based pose tracking methods are widely used in actual space missions due to their strong robustness to illumination and noise, high accuracy, and a wide range of target types. Some researchers determine the spacecraft pose by minimizing the distance from the edge extracted from the depth map of the spacecraft's three-dimensional model to the edge of the image, but this method lacks robustness to image edge noise. In addition, during the pose tracking process, rendering the spacecraft depth map is very time-consuming. To this end, many researchers use GPUs or FPGAs to alleviate computational time, but this increases the hardware cost of the algorithm and requires high energy consumption, making it unsuitable for actual space mission scenarios.

[0004] In addition, a considerable number of researchers have noted that using predefined geometric features on the target can significantly improve the accuracy and robustness of pose estimation. Some methods estimate pose by fitting image edges to recover 3D circles and 3D lines, but this is inefficient, prone to fitting errors, and susceptible to occlusion. Other methods address the real-time pose tracking of spacecraft by aligning the projections of manually predefined 3D circles, 3D lines, and quadratic surfaces on the spacecraft with its image edges. However, this method is not robust to occlusion and cannot handle spacecraft without line, cone, or quadratic surface features.

[0005] Therefore, existing pose tracking methods have problems such as poor universality, low efficiency, low pose estimation accuracy, and insufficient robustness to illumination and occlusion. Summary of the Invention

[0006] The present application provides a spacecraft posture tracking method based on monocular vision and three-dimensional geometric features to solve the problems in the prior art, such as poor universality, low efficiency, low posture estimation accuracy, and insufficient robustness to illumination, occlusion, etc. of the existing posture tracking methods.

[0007] Correspondingly, the present application also provides a spacecraft posture tracking system based on monocular vision and three-dimensional geometric features, an electronic device, and a computer-readable storage medium to ensure the implementation and application of the above method.

[0008] In order to solve the above technical problems, the present application discloses a spacecraft posture tracking method based on monocular vision and three-dimensional geometric features, the method comprising:

[0009] Extracting 3D geometric features from a pre-built 3D spacecraft model; the 3D geometric features include 3D edges and 3D contours;

[0010] Using trigonometric polynomials to approximate each 3D geometric feature with arbitrary precision, the analytical parametric equation of the 3D geometric feature is obtained;

[0011] The control points are sampled based on the analytical parametric equations of the 3D geometric features, and a matching relationship is established between the control points and the image edge points; the image edge points are extracted from the monocular image of the spacecraft;

[0012] The maximum likelihood estimate of the spacecraft pose is obtained by minimizing the ghosting error between the plane projection of the control points and the corresponding image edge points.

[0013] The maximum likelihood estimate of the spacecraft's attitude is corrected using an extended Kalman filter model based on second-order autoregression to obtain the final estimate of the spacecraft's attitude.

[0014] Preferably, the spacecraft three-dimensional model is composed of triangular mesh surfaces; extracting three-dimensional geometric features from the pre-built spacecraft three-dimensional model includes:

[0015] Extracting all sharp surface segments from the three-dimensional model of the spacecraft, and determining the three-dimensional edge of the spacecraft based on the sharp surface segments;

[0016] Sparse discrete points are uniformly sampled on all surface segments except the surface segments corresponding to the three-dimensional edges, and a three-dimensional outline of the spacecraft is generated based on the surface segments corresponding to all discrete points that meet preset conditions;

[0017] Wherein, a face segment refers to the edge of a triangular mesh face; a sharp face segment refers to the common edge of two adjacent triangular mesh face patches when the included angle between the two adjacent triangular mesh face patches is less than the angle threshold;

[0018] The preset conditions are that the discrete points can be obtained first, and the back-projection ray of the discrete points is tangent to the three-dimensional model of the spacecraft.

[0019] Preferably, extracting all sharp surface segments from the three-dimensional model of the spacecraft and determining the three-dimensional edge of the spacecraft based on the sharp surface segments includes:

[0020] Extract all sharp surface segments from the spacecraft three-dimensional model to generate a set of sharp surface segments;

[0021] A sharp face segment is randomly selected from the set of sharp face segments as a seed of a three-dimensional edge;

[0022] If there is only one unselected sharp face segment in the set of sharp face segments that has the same endpoint as the seed, then the sharp face segment is connected to the seed to extend the current seed;

[0023] The expanded seed is used as a new seed, and the expansion process is repeated until the seed is closed or no sharp surface segment that can be connected to the seed is found in the set of sharp surface segments, and the seed growth is determined to be complete;

[0024] When the grown seed meets the edge judgment condition, the seed is determined to be a three-dimensional edge of the spacecraft;

[0025] The edge judgment condition is that the grown seed is located outside the three-dimensional model of the spacecraft and the length of the seed is greater than or equal to the length threshold.

[0026] Preferably, when the grown seed meets the edge judgment condition, after determining the seed as a three-dimensional edge of the spacecraft, the method further includes:

[0027] If there are still unselected sharp surface segments in the set of sharp surface segments, another one or more three-dimensional edges of the spacecraft are generated according to the unselected sharp surface segments.

[0028] Preferably, sampling control points based on the analytical parametric equation of the three-dimensional geometric features and establishing a matching relationship between the control points and the edge points of the image include:

[0029] On the analytical parametric equation of each 3D geometric feature, control points are sampled within a specific range;

[0030] The image edge is extracted from the spacecraft monocular image according to the Canny operator. The image edge is composed of image edge points.

[0031] A matching relationship is formed between the control points and the image edge points at the same moment.

[0032] Preferably, the maximum likelihood estimate of the spacecraft pose is obtained by minimizing the ghosting error between the planar projection of the control point and the corresponding image edge point, including:

[0033] In any pose, the projection of the control point on the image plane is represented by a pinhole camera model;

[0034] Establishing an error relationship for calculating the ghosting error between the projection of the control point on the image plane and the corresponding image edge point;

[0035] The M-estimator is introduced to correct the ghosting error in the error relation and obtain the objective function of the ghosting error.

[0036] The objective function is optimized by iterative least squares method, and the maximum likelihood estimate of the spacecraft posture is obtained by solving it;

[0037] Among them, the objective function of ghosting error is expressed as:

[0038]

[0039] Where σ represents the ghosting error d j ε (l j ε ,p) standard deviation, {d j ε (l j ε ,p)} is considered to obey the normal independent and identical distribution with zero expectation; the initial value p of the pose p (0) =p k 0 ,{l j ε}The initial value of {l j ε(0)} corresponds to the arc length parameter of the visible control point that matches the image edge point; σ is set to {d j ε (l j ε(0) ,p (0) )}:

[0040]

[0041] Preferably, the maximum likelihood estimate of the spacecraft pose is corrected using an extended Kalman filter model based on second-order autoregression to obtain a final estimate of the spacecraft pose, including:

[0042] Define the state variable of the extended Kalman filter model as s k =[p k T p k-1 T ] T , where p -1 =p0,p k Represents the pose vector at time k;

[0043] Determine the state equation of the extended Kalman filter model:

[0044] p k+1 =log SE(3) (T k T k-1 -1 T k )+Γ k

[0045] p k =p k

[0046] Where, T = exp(p ^ ) is the pose vector p k The rigid body motion matrix generated in; p = log SE(3) (T) represents the pose vector p inversely solved from the motion matrix T; Γ k represents the prediction noise vector;

[0047] Determine the observation equation of the extended Kalman filter model:

[0048] z k =[I6 O6]s k +ε k

[0049] Where O6 represents the sixth-order zero matrix, ε k represents the observation noise vector, z k is the observed quantity;

[0050] Solve the extended Kalman filter model to obtain the final estimate of the spacecraft pose at the current moment.

[0051] The present application also discloses a spacecraft posture tracking system based on monocular vision and three-dimensional geometric features, the system comprising:

[0052] A geometric feature extraction module is used to extract 3D geometric features from a pre-built 3D spacecraft model; the 3D geometric features include 3D edges and 3D contours;

[0053] A trigonometric polynomial approximation module is used to use trigonometric polynomials to approximate each 3D geometric feature with arbitrary precision and obtain the analytical parametric equation of the 3D geometric feature;

[0054] A real-time pose estimation module is used to sample control points based on analytical parametric equations of 3D geometric features and establish matching relationships between control points and image edge points; image edge points are extracted from the spacecraft monocular image;

[0055] The real-time pose estimation module is also used to obtain the maximum likelihood estimate of the spacecraft pose by minimizing the ghosting error between the planar projection of the control points and the corresponding image edge points;

[0056] The pose correction module is used to correct the maximum likelihood estimate of the spacecraft pose using an extended Kalman filter model based on second-order autoregression to obtain the final estimate of the spacecraft pose.

[0057] The present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, one or more methods described in the present application are implemented.

[0058] The present application also discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, one or more methods described in the present application are implemented.

[0059] In this application, three-dimensional geometric features, including three-dimensional edges and three-dimensional contours, are extracted from a pre-built three-dimensional spacecraft model. Trigonometric polynomials are then used to approximate each three-dimensional geometric feature with arbitrary precision. The projection of the three-dimensional geometric features extracted by this method contains almost all the two-dimensional geometric information of the spacecraft image, and is therefore highly robust to shadow occlusion, lighting conditions, and changes in viewing angle. Furthermore, the acquisition of three-dimensional geometric features does not require GPU or FPGA rendering, thus ensuring its real-time and economical nature. Control points are then sampled based on the analytical parametric equations of the three-dimensional geometric features, and a matching relationship is established between the control points and the image edge points. The maximum likelihood estimate of the spacecraft pose is obtained by minimizing the ghosting error between the planar projection of the control points and the corresponding image edge points. Finally, the maximum likelihood estimate of the spacecraft pose is corrected using an extended Kalman filter model based on second-order autoregression to obtain the final estimate of the spacecraft pose, which can improve the efficiency and accuracy of spacecraft tracking. Therefore, the present application can achieve high-precision and high-efficiency spacecraft tracking at a very low cost, and is highly robust to shadow occlusion, different backgrounds, distance changes, and image noise.

[0060] In addition, the posture optimization algorithm in this application needs to optimize both the posture and arc length parameters, which improves the upper limit of the algorithm's accuracy.

[0061] Additional aspects and advantages of the present application will be given in the following description, which will become apparent from the following description, or will be understood through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0063] Figure 1A flowchart of a spacecraft posture tracking method based on monocular vision and three-dimensional geometric features provided in an embodiment of the present application;

[0064] Figure 2 An overall flow chart of the spacecraft posture tracking method provided in an embodiment of the present application;

[0065] Figure 3 A schematic diagram of the structure of a spacecraft posture tracking system based on monocular vision and three-dimensional geometric features provided in an embodiment of the present application;

[0066] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following describes embodiments of the present application in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0068] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of this application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0069] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless specifically defined as herein, will not be interpreted in an idealized or overly formal sense.

[0070] The solution provided in the embodiments of the present application can be executed by any electronic device, such as a terminal device or a server, wherein the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to these. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions on this. With respect to the technical problems existing in the prior art, the spacecraft posture tracking method and system based on monocular vision and three-dimensional geometric features provided in this application is intended to solve at least one of the technical problems of the prior art.

[0071] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0072] The present application embodiment provides a possible implementation method, such as Figure 1 As shown, a flowchart of a spacecraft posture tracking method based on monocular vision and three-dimensional geometric features is provided. The scheme can be executed by any electronic device, and optionally, can be executed on the server side or terminal device.

[0073] like Figure 1 As shown in , the method may include the following steps:

[0074] Step 101: extract three-dimensional geometric features from a pre-built three-dimensional spacecraft model; the three-dimensional geometric features include three-dimensional edges and three-dimensional contours.

[0075] In the embodiments of the present application, three-dimensional geometric features of arbitrary shapes can be automatically extracted. Three-dimensional edges can be extracted offline, while three-dimensional contours change as the camera's viewing angle changes, necessitating online extraction of the contours. The extracted three-dimensional edges and contours contain nearly all the geometric information of the spacecraft image, making them highly robust to shadows, lighting conditions, and changes in viewing angle. Furthermore, by utilizing the structure of the spacecraft's three-dimensional model to extract the three-dimensional edges and contours, the algorithm has low hardware cost and energy consumption.

[0076] Step 102: Approximate each three-dimensional geometric feature with arbitrary precision using trigonometric polynomials to obtain an analytical parametric equation of the three-dimensional geometric feature.

[0077] Performing arbitrary precision trigonometric polynomial approximation on three-dimensional geometric features can improve the accuracy of three-dimensional geometric features and provide an accurate basis for subsequent spacecraft posture tracking.

[0078] Step 103: sample control points based on the analytical parametric equation of the three-dimensional geometric features, and establish a matching relationship between the control points and the image edge points; the image edge points are extracted from the monocular image of the spacecraft.

[0079] Step 104 , obtaining a maximum likelihood estimate of the spacecraft pose by minimizing the ghosting error between the planar projection of the control point and the corresponding image edge point.

[0080] In the embodiment of the present application, real-time pose estimation based on the SE (3) group is adopted. First, the plane projection of the control point is determined. In the embodiment of the present application, the plane projection of different control points can be represented by a unified expression. Then, a relationship between the plane projection of the control point and the corresponding image edge point is established. Finally, an optimization algorithm is used to optimize the ghosting error between the plane projection of the control point and the corresponding image edge point to obtain the maximum likelihood estimate of the spacecraft pose.

[0081] Step 105 , correcting the maximum likelihood estimate of the spacecraft's position and attitude using an extended Kalman filter model based on second-order autoregression to obtain a final estimate of the spacecraft's position and attitude.

[0082] The extended Kalman filter model is used to further correct the maximum likelihood estimate of the spacecraft pose, which can make the pose estimation more accurate and robust.

[0083] In an embodiment of the present application, three-dimensional geometric features including three-dimensional edges and three-dimensional contours are extracted from a pre-built three-dimensional model of the spacecraft, and then trigonometric polynomials are used to approximate each three-dimensional geometric feature with arbitrary precision. The projection of the three-dimensional geometric features extracted by this method contains almost all the two-dimensional geometric information of the spacecraft image, and is therefore highly robust to shadow occlusion, lighting conditions, and changes in viewing angle. In addition, the acquisition of the three-dimensional geometric features does not require GPU or FPGA rendering, thus ensuring its real-time and economical nature. Subsequently, control points are sampled based on the analytical parametric equations of the three-dimensional geometric features, and a matching relationship between the control points and the image edge points is established. By minimizing the ghosting error between the planar projection of the control points and the corresponding image edge points, the maximum likelihood estimate of the spacecraft pose is obtained. Finally, the maximum likelihood estimate of the spacecraft pose is corrected using an extended Kalman filter model based on second-order autoregression to obtain the final estimate of the spacecraft pose, which can improve the efficiency and accuracy of spacecraft tracking. Therefore, the embodiments of the present application can achieve high-precision and high-efficiency spacecraft tracking at a very low cost, and are highly robust to shadow occlusion, different backgrounds, distance changes, and image noise.

[0084] In an optional embodiment, the spacecraft three-dimensional model is composed of triangular mesh patches; extracting three-dimensional geometric features from a pre-built spacecraft three-dimensional model includes:

[0085] Extract all sharp face segments from the 3D spacecraft model and determine the 3D edges of the spacecraft based on the sharp face segments. A sharp face segment is defined as the common edge between two adjacent triangular mesh facets when the angle between them is less than a threshold.

[0086] Sparse discrete points are uniformly sampled on all surface segments except those corresponding to three-dimensional edges, and the three-dimensional outline of the spacecraft is generated based on the surface segments corresponding to all discrete points that meet preset conditions; where the surface segments refer to the edges of the triangular mesh patches; the preset condition is that the back-projection ray of the discrete points is tangent to the three-dimensional model of the spacecraft.

[0087] For example, the angle threshold can be empirically set to 120°. When the angle between two adjacent triangular mesh facets is less than 120°, the shared edge of these two adjacent triangular mesh facets is considered a sharp face segment. Based on this judgment criterion, embodiments of the present application are used to extract three-dimensional spacecraft edges composed of these sharp face segments from a three-dimensional spacecraft model. Sharp face segments with the same endpoints can be connected to form a single three-dimensional edge, and in embodiments of the present application, at least one three-dimensional edge can be extracted.

[0088] In addition to extracting the three-dimensional edges of the spacecraft, the embodiment of the present application also extracts the three-dimensional contour of the spacecraft. Unlike the three-dimensional edges extracted offline, the three-dimensional contour of the spacecraft will change with the camera viewing angle. Therefore, when running the pose tracking algorithm, the three-dimensional contour must be extracted online. In order to ensure tracking efficiency, the embodiment of the present application only considers the three-dimensional contour portion that does not overlap with the three-dimensional edge on the three-dimensional model of the spacecraft. In order to be able to detect the three-dimensional contour of the spacecraft online, the embodiment of the present application first uniformly samples sparse discrete points offline on each face segment that does not belong to any extracted three-dimensional edge. At each moment k of pose tracking, the initial pose p predicted by the extended Kalman filter model (EKF) at that moment is used. k0 If a sampling point satisfies the conditions of visibility (visibility can be quickly determined by ray casting) and its back-projected ray is tangent to the spacecraft 3D model, then the sampling point is considered a contour point. In this embodiment of the application, the surface segment where the contour point is located is called a 3D contour segment, and all three-dimensional contour segments are combined to form a 3D contour.

[0089] The 3D geometric features in the embodiments of this application are not limited to manually predefined 3D circles, 3D lines, and quadratic surfaces. 3D geometric features (3D edges and 3D contours) of any shape can be automatically extracted. The projection of the extracted 3D geometric features contains nearly all the 2D geometric information of the spacecraft image, making them highly robust to shadow occlusion, lighting conditions, and changes in viewing angle. Furthermore, the acquisition of 3D geometric features does not require GPU or FPGA rendering, thus ensuring real-time and cost-effectiveness.

[0090] In an optional embodiment, extracting all sharp surface segments from the three-dimensional spacecraft model and determining the three-dimensional edge of the spacecraft based on the sharp surface segments includes:

[0091] Extract all sharp surface segments from the spacecraft three-dimensional model to generate a set of sharp surface segments;

[0092] Selecting a sharp face segment from a set of sharp face segments as a seed of a three-dimensional edge;

[0093] If there is only one unselected sharp face segment in the set of sharp face segments that has the same endpoint as the seed, then the sharp face segment is connected to the seed to extend the current seed;

[0094] The expanded seed is used as a new seed, and the expansion process is repeated until the seed is closed or no sharp surface segment that can be connected to the seed is found in the set of sharp surface segments, and the seed growth is determined to be complete;

[0095] When the grown seed meets the edge judgment condition, the seed is determined to be a three-dimensional edge of the spacecraft;

[0096] The edge judgment condition is that the grown seed is located outside the three-dimensional model of the spacecraft and the length of the seed is greater than or equal to the length threshold.

[0097] In an embodiment of the present application, let G be a set containing all sharp surface fragments, wherein each sharp surface fragment is associated with a binary state indicator variable, and the indicator variables of all sharp surface fragments are set to 0 in the initial state, indicating an unprocessed state. First, the algorithm randomly selects a sharp surface fragment from the set G as the seed of a three-dimensional edge, and sets its indicator variable to 1, marking it as processed. If there is only one unprocessed (i.e., the indicator variable is 0) sharp surface fragment in the set G that has the same endpoint as the seed, then the sharp surface fragment is connected to the seed to extend the current seed, and its indicator variable is set to 1. The expanded seed is used as the new seed, and this extension process is repeated until the seed is closed or no sharp surface fragment that can be connected to the seed can be found in the set G. After the seed growth is completed, in the embodiment of the present application, a multi-perspective ray casting method is used to determine whether the seed is located outside the three-dimensional model. If the seed is located outside and its length is greater than or equal to the preset length threshold, it is determined to be a three-dimensional edge of the spacecraft.

[0098] In an optional embodiment, when the grown seed meets the edge judgment condition, after determining the seed as a three-dimensional edge of the spacecraft, the method further includes:

[0099] If there are still unselected sharp surface segments in the set of sharp surface segments, another one or more three-dimensional edges of the spacecraft are generated according to the unselected sharp surface segments.

[0100] In the embodiment of the present application, if there are still unprocessed sharp surface segments in the set G, new seeds are randomly selected therefrom, and the above seed growth and judgment process is repeated until all sharp surface segments are processed.

[0101] In an optional embodiment, trigonometric polynomials are used to approximate each three-dimensional geometric feature with arbitrary precision to obtain an analytical parametric equation of the three-dimensional geometric feature. The specific implementation is as follows:

[0102] Assume that n b Three-dimensional geometric features, l is the arc length parameter, then each three-dimensional geometric feature w i (l),i=1,2,…,n b All of them can be approximated by trigonometric polynomials to obtain the following smooth and continuously differentiable analytical parametric equations:

[0103]

[0104] Where a h and b h is the three-dimensional coefficient vector of the polynomial, and T is called the period.

[0105] In an optional embodiment, sampling control points based on the analytical parametric equation of the three-dimensional geometric features and establishing a matching relationship between the control points and the edge points of the image include:

[0106] On the analytical parametric equation of each 3D geometric feature, control points are sampled within a specific range;

[0107] The image edge is extracted from the spacecraft monocular image according to the Canny operator. The image edge is composed of image edge points.

[0108] A matching relationship is formed between the control points and the image edge points at the same moment.

[0109] In the embodiment of the present application, real-time pose estimation based on SE(3) group is used. Specifically:

[0110] In every trigonometric polynomial F i (l) on, in The control points are sampled according to the following rules within the range:

[0111]

[0112] Here, δ is the step size parameter.

[0113] Then, the Canny operator is used to extract the image edge from the spacecraft monocular image, and a matching relationship is formed between the control points and the image edge.

[0114] In an optional embodiment, obtaining a maximum likelihood estimate of the spacecraft pose by minimizing the ghosting error between the planar projection of the control point and the corresponding image edge point includes:

[0115] In any pose, the projection of the control point on the image plane is represented by a pinhole camera model;

[0116] Establishing an error relationship for calculating the ghosting error between the projection of the control point on the image plane and the corresponding image edge point;

[0117] The M-estimator is introduced to correct the ghosting error in the error relation and obtain the objective function of the ghosting error.

[0118] The objective function is optimized by iterative least squares method, and the maximum likelihood estimate of the spacecraft pose is obtained.

[0119] Among them, the objective function of ghosting error is expressed as:

[0120]

[0121] Where σ represents the ghosting error d j ε (lj ε ,p) standard deviation, {d j ε (l j ε ,p)} is considered to obey the normal independent and identical distribution with zero expectation; the initial value p of the pose p (0) =p k 0 ,{l j ε}The initial value of {l j ε(0)} corresponds to the arc length parameter of the visible control point that matches the edge point of the image; in order to improve the efficiency of the algorithm, σ is set to {d j ε (l j ε(0) ,p (0) )}:

[0122]

[0123] In the embodiment of the present application, let A be the set of all extracted three-dimensional geometric features. At time k, assume that the three-dimensional geometric feature F ε (l),ε∈A is finally matched to the edge point of the two-dimensional image {x j ε ,j=1,2,…,n ε}, where n ε Indicates F ε (l) The number of matched image edge points. Under pose p, F ε (l) The projection on the image plane can be expressed as follows using the pinhole camera model:

[0124]

[0125] Among them, K (1,2) represents the first two rows of matrix K, Indicates F ε The homogeneous vector of (l), h3 = [0 0 1 0] T So, a ε (lj ε ,p) and x j ε The reprojection error between can be calculated by the following formula:

[0126]

[0127] In order to reduce the influence of outliers on pose optimization, the embodiment of the present application introduces an M-estimator to correct the reprojection error in formula (6). Therefore, the pose p can be obtained by minimizing the sum of all scaled reprojection error M-estimators to obtain the above formula (3).

[0128] The M estimator in the embodiment of the present application uses the Tukey estimator. The optimization problem of formula (3) can be solved by iterative least squares (IRLS). At each iteration time s (s ≥ 0), IRLS will solve the following sub-problems corresponding to formula (3):

[0129]

[0130] Among them, d j ε(s) =d j ε (l j ε(s) ,p (s) ), ω(x) = ρ′(x) / x is the weight function, which has the following form:

[0131]

[0132] The Newton method can be used to solve the objective function expressed in formula (7), and the motion matrix T expressed by the SE (3) group at the current time k can be obtained. k . Then you can go to T k Inversely solve the pose vector p at the current k moment k , and it is the maximum likelihood estimate.

[0133] The posture optimization algorithm in the embodiment of the present application needs to optimize the posture and arc length parameters at the same time, which improves the upper limit of the algorithm's accuracy.

[0134] In an optional embodiment, a maximum likelihood estimate of the spacecraft pose is corrected using an extended Kalman filter model based on a second-order autoregressive method to obtain a final estimate of the spacecraft pose, including:

[0135] Define the state variable of the extended Kalman filter model as s k =[p k T p k-1 T ] T , where p -1 =p0,p k Represents the pose vector at time k;

[0136] Determine the state equation of the extended Kalman filter model:

[0137]

[0138] Where, T = exp(p ^ ) is the pose vector p k The rigid body motion matrix generated in; p = log SE(3) (T) represents the pose vector p obtained by inversely solving the motion matrix in T; Γ k represents the prediction noise vector;

[0139] Determine the observation equation of the extended Kalman filter model:

[0140] z k =[I6 O6]s k +ε k (10)

[0141] Where O6 represents the sixth-order zero matrix, ε k represents the observation noise vector, z k is the observed quantity;

[0142] Solve the extended Kalman filter model to obtain the current k-time pose vector p k The maximum a posteriori probability estimate of , that is, the final estimate of the spacecraft pose.

[0143] The posture parameters finally obtained in the embodiment of the present application are maximum a posteriori probability estimates, which have higher accuracy and robustness.

[0144] like Figure 2 As shown, based on the above method, the process of implementing spacecraft posture tracking in this application includes:

[0145] (1) 3D edge extraction and mathematical expression, including automatic extraction of 3D edges and trigonometric polynomial approximation of 3D edges.

[0146] (2) 3D contour extraction and mathematical expression, including automatic 3D contour extraction and trigonometric polynomial approximation of 3D contour. 3D contour extraction is performed after 3D edge extraction, based on each face segment that does not belong to any extracted 3D edge. 3D contour extraction requires the input spacecraft image I k The current moment k is judged. If k>0, the automatic extraction of the three-dimensional contour is performed directly; otherwise, the current pose of the spacecraft is initialized using the preset pose initialization algorithm, and then the automatic extraction of the three-dimensional contour is performed.

[0147] (3) Real-time pose determination, including image edge matching of 3D geometric features and SE(3) pose optimization based on M estimation and Newton method to obtain the maximum likelihood estimate of the spacecraft pose.

[0148] (4) Extended Kalman filter (EKF) based on SE (3). When the spacecraft is at the current moment k>0, the extended Kalman filter model is used to correct the maximum likelihood estimate and output the final three-dimensional pose p k ; Then the state equation of the extended Kalman filter model is used to predict the initial position of the spacecraft at the next moment.

[0149] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application also provides a spacecraft posture tracking system based on monocular vision and three-dimensional geometric features, such as Figure 3 As shown, the system includes:

[0150] The geometric feature extraction module 301 is used to extract 3D geometric features from a pre-built 3D model of a spacecraft; the 3D geometric features include 3D edges and 3D contours;

[0151] A trigonometric polynomial approximation module 302 is configured to approximate each three-dimensional geometric feature with arbitrary precision using trigonometric polynomials to obtain an analytical parametric equation of the three-dimensional geometric feature;

[0152] The real-time pose estimation module 303 is used to sample control points based on the analytical parametric equations of the three-dimensional geometric features and establish a matching relationship between the control points and the image edge points; the image edge points are extracted from the monocular image of the spacecraft;

[0153] The real-time pose estimation module 303 is further configured to obtain a maximum likelihood estimate of the spacecraft pose by minimizing the ghosting error between the planar projections of the control points and the corresponding image edge points;

[0154] The posture correction module 304 is used to correct the maximum likelihood estimate of the spacecraft posture using an extended Kalman filter model based on second-order autoregression to obtain a final estimate of the spacecraft posture.

[0155] In an embodiment of the present application, three-dimensional geometric features including three-dimensional edges and three-dimensional contours are extracted from a pre-built three-dimensional model of the spacecraft, and then trigonometric polynomials are used to approximate each three-dimensional geometric feature with arbitrary precision. The projection of the three-dimensional geometric features extracted by this method contains almost all the two-dimensional geometric information of the spacecraft image, and is therefore highly robust to shadow occlusion, lighting conditions, and changes in viewing angle. In addition, the acquisition of the three-dimensional geometric features does not require GPU or FPGA rendering, thus ensuring its real-time and economical nature. Subsequently, control points are sampled based on the analytical parametric equations of the three-dimensional geometric features, and a matching relationship between the control points and the image edge points is established. By minimizing the ghosting error between the planar projection of the control points and the corresponding image edge points, the maximum likelihood estimate of the spacecraft pose is obtained. Finally, the maximum likelihood estimate of the spacecraft pose is corrected using an extended Kalman filter model based on second-order autoregression to obtain the final estimate of the spacecraft pose, which can improve the efficiency and accuracy of spacecraft tracking. Therefore, the embodiments of the present application can achieve high-precision and high-efficiency spacecraft tracking at a very low cost, and are highly robust to shadow occlusion, different backgrounds, distance changes, and image noise.

[0156] The spacecraft posture tracking system based on monocular vision and three-dimensional geometric features provided in the embodiment of the present application can achieve Figure 1 and Figure 2 To avoid repetition, the various processes implemented in the method embodiment will not be described again here.

[0157] The spacecraft posture tracking system based on monocular vision and three-dimensional geometric features in the embodiment of the present application can execute the spacecraft posture tracking method based on monocular vision and three-dimensional geometric features provided in the embodiment of the present application, and its implementation principle is similar. The actions performed by each module and unit in the spacecraft posture tracking system based on monocular vision and three-dimensional geometric features in each embodiment of the present application correspond to the steps in the spacecraft posture tracking method based on monocular vision and three-dimensional geometric features in each embodiment of the present application. For the detailed functional description of each module of the spacecraft posture tracking system based on monocular vision and three-dimensional geometric features, please refer to the description of the corresponding spacecraft posture tracking method based on monocular vision and three-dimensional geometric features shown in the previous text, which will not be repeated here.

[0158] Based on the same principles as the methods shown in the embodiments of this application, the embodiments of this application also provide an electronic device, which may include but is not limited to: a processor and a memory; the memory is used to store a computer program; and the processor is used to execute the spacecraft posture tracking method based on monocular vision and three-dimensional geometric features shown in any optional embodiment of this application by calling the computer program. Compared with the existing technology, the spacecraft posture tracking method based on monocular vision and three-dimensional geometric features provided in this application can achieve high-precision and high-efficiency spacecraft tracking at a very low cost, and is highly robust to shadow occlusion, different backgrounds, distance changes, and image noise.

[0159] In an optional embodiment, an electronic device is also provided, such as Figure 4 As shown, Figure 4 The electronic device 400 shown may be a server, including a processor 401 and a memory 403. The processor 401 and the memory 403 are connected, for example, via a bus 402. Optionally, the electronic device 400 may further include a transceiver 404. It should be noted that in actual applications, the number of transceivers 404 is not limited to one, and the structure of the electronic device 400 does not constitute a limitation on the embodiments of the present application.

[0160] Processor 401 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 401 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0161] The bus 402 may include a path for transmitting information between the above components. The bus 402 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 402 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0162] The memory 403 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0163] The memory 403 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 401. The processor 401 is used to execute the application code stored in the memory 403 to implement the content shown in the above method embodiment.

[0164] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0165] The server provided in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to these. The terminal and the server can be directly or indirectly connected through wired or wireless communication, and this application does not limit this.

[0166] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0167] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0168] It should be noted that the computer-readable storage medium mentioned above in this application may also be a computer-readable signal medium or a combination of a computer-readable storage medium and a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0169] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0170] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0171] According to one aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to implement the spacecraft pose tracking method and system based on monocular vision and three-dimensional geometric features provided in the various optional implementations described above.

[0172] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0173] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0174] The modules described in the embodiments of this application may be implemented in software or hardware. In some cases, the name of a module does not limit the module itself. For example, a geometric feature extraction module may also be described as a "geometric feature extraction module for extracting three-dimensional geometric features from a pre-built three-dimensional spacecraft model."

[0175] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A spacecraft posture tracking method based on monocular vision and three-dimensional geometric features, characterized in that: The method comprises: Extracting three-dimensional geometric features from a pre-built three-dimensional model of a spacecraft; the three-dimensional geometric features include three-dimensional edges and three-dimensional contours; Using trigonometric polynomials to approximate each 3D geometric feature with arbitrary precision, the analytical parametric equation of the 3D geometric feature is obtained; Sampling control points based on analytical parametric equations of three-dimensional geometric features and establishing matching relationships between the control points and image edge points; the image edge points are extracted from a monocular image of a spacecraft; The maximum likelihood estimate of the spacecraft pose is obtained by minimizing the ghosting error between the plane projection of the control points and the corresponding image edge points. The maximum likelihood estimate of the spacecraft's attitude is corrected using an extended Kalman filter model based on second-order autoregression to obtain the final estimate of the spacecraft's attitude.

2. The spacecraft posture tracking method based on monocular vision and three-dimensional geometric features according to claim 1 is characterized in that: The spacecraft three-dimensional model is composed of triangular mesh surfaces; the three-dimensional geometric features are extracted from the pre-built spacecraft three-dimensional model; including: extracting all sharp surface segments from the three-dimensional model of the spacecraft, and determining the three-dimensional edge of the spacecraft based on the sharp surface segments; Uniformly sampling sparse discrete points on all surface segments except the surface segments corresponding to the three-dimensional edges, and generating a three-dimensional contour of the spacecraft based on the surface segments corresponding to all the discrete points that meet preset conditions; The face segment refers to the edge of the triangular mesh face; the sharp face segment refers to the common edge of two adjacent triangular mesh face when the included angle between the two adjacent triangular mesh face is less than the angle threshold; The preset condition is that the discrete points can be located first, and the back-projection ray of the discrete points is tangent to the three-dimensional model of the spacecraft.

3. The spacecraft posture tracking method based on monocular vision and three-dimensional geometric features according to claim 2, characterized in that: Extracting all sharp surface segments from the three-dimensional model of the spacecraft and determining the three-dimensional edge of the spacecraft based on the sharp surface segments includes: extracting all sharp surface segments from the three-dimensional model of the spacecraft to generate a set of sharp surface segments; Randomly selecting a sharp face segment from the set of sharp face segments as a seed of a three-dimensional edge; If there is only one unselected sharp surface segment in the set of sharp surface segments that has the same endpoint as the seed, then the sharp surface segment is connected to the seed to extend the current seed; The expanded seed is used as a new seed, and the expansion process is repeated until the seed is closed or no sharp surface segment that can be connected to the seed is found in the set of sharp surface segments, and the seed growth is determined to be complete; When the grown seed meets the edge judgment condition, the seed is determined to be a three-dimensional edge of the spacecraft; The edge judgment condition is that the grown seed is located outside the three-dimensional model of the spacecraft and the length of the seed is greater than or equal to a length threshold.

4. The spacecraft posture tracking method based on monocular vision and three-dimensional geometric features according to claim 3 is characterized in that: When the grown seed satisfies the edge judgment condition, after determining the seed as a three-dimensional edge of the spacecraft, the method further includes: If there are still unselected sharp surface segments in the set of sharp surface segments, another one or more three-dimensional edges of the spacecraft are generated according to the unselected sharp surface segments.

5. The spacecraft posture tracking method based on monocular vision and three-dimensional geometric features according to claim 1, characterized in that: The sampling of control points based on the analytical parametric equation for the three-dimensional geometric features and the establishment of a matching relationship between the control points and the edge points of the image include: On the analytical parametric equation of each 3D geometric feature, control points are sampled within a specific range; Extracting image edges from a spacecraft monocular image according to a Canny operator, wherein the image edges are composed of image edge points; A matching relationship is formed between the control points and the image edge points at the same moment.

6. The spacecraft posture tracking method based on monocular vision and three-dimensional geometric features according to claim 1, characterized in that: Obtaining a maximum likelihood estimate of the spacecraft pose by minimizing the ghosting error between the planar projection of the control point and the corresponding image edge point includes: In any pose, the projection of the control point on the image plane is represented by a pinhole camera model; Establishing an error relationship for calculating the ghosting error between the projection of the control point on the image plane and the corresponding image edge point; Introducing an M estimator to correct the ghosting error in the error relation to obtain a target function of the ghosting error; The objective function is optimized by iterative least squares method, and the maximum likelihood estimate of the spacecraft posture is obtained by solving.

7. The spacecraft posture tracking method based on monocular vision and three-dimensional geometric features according to claim 1, characterized in that: The method of correcting the maximum likelihood estimate of the spacecraft's attitude using the extended Kalman filter model based on second-order autoregression to obtain a final estimate of the spacecraft's attitude includes: Define the state variable of the extended Kalman filter model as s k =[p k T p k-1 T ] T , where p -1 =p0,p k Represents the pose vector at time k; Determine the state equation of the extended Kalman filter model: p k+1 =log SE(3) (T k T k-1 -1 T k )+Γ k p k =p k Where, T = exp(p ^ ) is the pose vector p k The rigid body motion matrix generated in; p = log SE(3) (T) represents the pose vector p inversely solved from the motion matrix T; Γ k represents the prediction noise vector; Determine the observation equation of the extended Kalman filter model: With k =[I6 O6]s k +ε k Where O6 represents the sixth-order zero matrix, ε k represents the observation noise vector, z k is the observed quantity; Solve the extended Kalman filter model to obtain the final estimate of the spacecraft pose at the current moment.

8. A spacecraft posture tracking system based on monocular vision and three-dimensional geometric features, characterized in that: The system comprises: A geometric feature extraction module is used to extract three-dimensional geometric features from a pre-built three-dimensional spacecraft model; the three-dimensional geometric features include three-dimensional edges and three-dimensional contours; A trigonometric polynomial approximation module is used to use trigonometric polynomials to approximate each 3D geometric feature with arbitrary precision and obtain the analytical parametric equation of the 3D geometric feature; A real-time pose estimation module is used to sample control points based on analytical parametric equations of 3D geometric features and establish a matching relationship between the control points and image edge points; the image edge points are extracted from the spacecraft monocular image; The real-time pose estimation module is also used to obtain the maximum likelihood estimate of the spacecraft pose by minimizing the ghosting error between the planar projection of the control points and the corresponding image edge points; The pose correction module is used to correct the maximum likelihood estimate of the spacecraft pose using an extended Kalman filter model based on second-order autoregression to obtain the final estimate of the spacecraft pose.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the method according to any one of claims 1 to 7 is implemented when the processor executes the program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images