VO reliability assessment methods, model training methods, devices, equipment and products

Abstract

This disclosure relates to a VO reliability assessment method, model training method, apparatus, device, and product. It involves acquiring an image sequence and its actual trajectory and pose, inputting the image sequence into a VO processing device, and obtaining the internal variables of the VO processing device during image sequence processing, as well as the estimated trajectory and pose of the acquisition device. The mounting angle error and relative pose error of the VO processing device when processing each image are determined using the actual trajectory and pose, and the estimated trajectory and pose. Based on these errors, a reliability label for each image is determined. Thus, a VO reliability assessment model is trained using the internal variables and reliability labels obtained from the VO processing device's image sequence processing. This disclosure improves the accuracy of VO reliability assessment based on the trained VO reliability assessment model.

Description

Technical Field

[0001] This disclosure relates to the field of AR navigation technology, and in particular to a VO reliability assessment method, model training method, apparatus, device, and product. Background Technology

[0002] Augmented Reality (AR) navigation technology integrates real-world visual information into the navigation interface. AR navigation provides users with an immersive navigation experience and improves positioning accuracy. Currently, a crucial positioning scheme in AR navigation is the fusion of Visual Odometry (VO) and Pedestrian Dead Reckoning (PDR). However, this scheme requires real-time reliability assessment of the VO, and the assessment results are input into modules such as PDR. Therefore, the accuracy of the assessment directly affects the accuracy of PDR and the final fusion positioning scheme. Consequently, improving the accuracy of VO reliability assessment is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides a VO reliability assessment method, model training method, apparatus, equipment, and product.

[0004] A first aspect of this disclosure provides a method for training a VO reliability assessment model. The method includes acquiring an image sequence and the true trajectory and true pose of an acquisition device during image acquisition; inputting the image sequence into a VO processing device for processing, acquiring internal variables of the VO processing device during image sequence processing, and the estimated trajectory and estimated pose of the acquisition device; determining the mounting angle error and relative pose error of the VO processing device when processing each image based on the true trajectory and true pose, the estimated trajectory and estimated pose; determining the reliability label of the VO processing device when processing each image based on the mounting angle error and relative pose error; and training a VO reliability assessment model based on the internal variables of the VO processing device during image sequence processing and the reliability label of the VO processing device when processing each image.

[0005] A second aspect of this disclosure provides a VO reliability assessment method, the method comprising: acquiring a first image to be processed; inputting the first image into a VO processing device, acquiring a first internal variable of the VO processing device during the processing of the first image; inputting the first internal variable and a second internal variable of the VO processing device during the processing of multiple second images into a VO reliability assessment model trained based on the method of the first aspect, to obtain a VO reliability assessment result of the VO processing device when processing the first image; wherein, the second image refers to an image acquired before the first image.

[0006] A third aspect of this disclosure provides a training apparatus for a VO reliability assessment model, the apparatus comprising:

[0007] The first acquisition module is used to acquire the image sequence and the real trajectory and real pose of the acquisition device when acquiring the image sequence;

[0008] The second acquisition module is used to input the image sequence into the VO processing device for processing, and to acquire the internal variables of the VO processing device during the processing of the image sequence, as well as the estimated trajectory and estimated pose of the acquisition device.

[0009] The first determining module is used to determine the installation angle error and relative pose error of the VO processing device when processing each image based on the real trajectory and the real pose, the estimated trajectory and the estimated pose;

[0010] The second determining module is used to determine the reliability label of the VO processing device when processing each image based on the installation angle error and relative pose error of the VO processing device when processing each image.

[0011] The training module is used to train a VO reliability assessment model based on the internal variables of the VO processing device during the processing of the image sequence and the reliability labels of the VO processing device when processing each image.

[0012] A fourth aspect of this disclosure provides a VO reliability assessment apparatus, comprising:

[0013] The first acquisition module is used to acquire the first image to be processed.

[0014] The second acquisition module is used to input the first image into the VO processing device and acquire the first internal variable of the VO processing device during the process of processing the first image.

[0015] The reliability assessment module is used to input the first internal variable and the second internal variable of the VO processing device in the process of processing multiple second images into the VO reliability assessment model trained based on the first aspect of the method, so as to obtain the VO reliability assessment result of the VO processing device when processing the first image; wherein, the second image refers to the image collected before the first image.

[0016] A fifth aspect of this disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, can implement the method described in the first aspect above.

[0017] A sixth aspect of this disclosure provides a terminal device, which includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, can implement the method described in the second aspect above.

[0018] A seventh aspect of this disclosure provides a computer program product stored in a storage medium, which, when run, can implement the methods described in the first or second aspect above.

[0019] An eighth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed, can implement the methods described in the first or second aspect.

[0020] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0021] This embodiment of the disclosure acquires image sequences and the actual trajectory and pose of the acquisition device during image sequence acquisition. The image sequences are then input into a VO processing device to obtain the internal variables of the VO processing device during image sequence processing, as well as the estimated trajectory and pose of the acquisition device. The mounting angle error and relative pose error of the VO processing device during image processing are determined using the actual trajectory and pose, and the estimated trajectory and pose. Based on these errors, a reliability label is determined for each image. Thus, a VO reliability assessment model is trained using the internal variables and reliability labels of the VO processing device during image sequence processing. This embodiment of the disclosure considers the dependence of downstream modules such as PDR on the mounting angle when training the VO reliability assessment model. By using the mounting angle error and relative pose error as the basis for label calculation, the accuracy of the reliability labels is improved. Therefore, training the VO reliability assessment model based on the reliability labels and the internal variables of the VO processing device during image sequence processing improves the accuracy of the VO reliability assessment model, thereby improving the accuracy of the VO reliability assessment. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of a model training scenario provided in an embodiment of this disclosure;

[0025] Figure 2 This is a flowchart of a training method for a VO reliability assessment model provided in an embodiment of this disclosure;

[0026] Figure 3 This is a flowchart of a method for determining the installation angle error of a VO processing device when processing images;

[0027] Figure 4 This is a flowchart of a method for determining the relative pose error of a VO processing device when processing images;

[0028] Figure 5 This is a flowchart of a VO reliability assessment method provided in an embodiment of this disclosure;

[0029] Figure 6 This is a schematic diagram of a VO reliability assessment scenario;

[0030] Figure 7 This is a schematic diagram of the structure of a training device for a VO reliability assessment model provided in an embodiment of this disclosure;

[0031] Figure 8 This is a schematic diagram of the structure of a VO reliability assessment device provided in an embodiment of this disclosure;

[0032] Figure 9 This is a schematic diagram of the structure of a terminal device according to an embodiment of this disclosure. Detailed Implementation

[0033] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0034] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0035] To better understand the technical solutions of the embodiments of this disclosure, some technical terms involved in the embodiments of this disclosure will be explained first.

[0036] A Visual Odometry (VO) processing device is a means of estimating the motion of a data acquisition device based on captured images. This device may include a front-end and a back-end. The front-end processes the input images using a visual odometry (VO) system to obtain the pose of the data acquisition device, thus performing pose tracking. The back-end uses a preset optimization algorithm to optimize the pose calculated by the front-end to obtain a globally consistent trajectory.

[0037] The installation angle of the data acquisition device refers to the angle between the direction of travel and the orientation of the data acquisition device.

[0038] PDR (Pedestrian Position Estimation) is a pedestrian pose estimation algorithm based on inertial sensors. PDR is inaccurate in estimating the mounting angle of the data acquisition device; therefore, in practical applications, techniques such as VO (Voice of Target) are often used to provide a more accurate mounting angle for PDR, thereby improving its accuracy.

[0039] Referring to the background art, improving the accuracy of VO reliability assessment is crucial for enhancing the positioning accuracy of VR navigation. Therefore, this disclosure provides a training scheme for a VO reliability assessment model to evaluate VO reliability using the trained model, thereby improving the accuracy of VO reliability assessment. For example, Figure 1This is a schematic diagram of a model training scenario provided in an embodiment of this disclosure. For example... Figure 1 As shown, in some embodiments, the image sequence acquired by the acquisition device can be input into the VO processing device for processing. This yields the internal variables of the VO processing device during the image sequence processing (including internal variables for processing each image, which include at least one of front-end processing-related feature variables, back-end optimization-related feature variables, and trajectory-related feature variables). The estimated trajectory and estimated pose of the acquisition device are then estimated by the VO processing device (each trajectory point corresponds to one image in the image sequence and one estimated pose). Then, using related techniques, based on the actual trajectory and actual pose of the acquisition device during image sequence acquisition, combined with the estimated trajectory and estimated pose obtained by the VO processing device, the installation angle error and relative pose error of the VO processing device when processing each image are determined. Then, data preprocessing is performed. For each image, the mounting angle error and relative pose error of the VO processing device when processing that image are compared with corresponding thresholds. If either the mounting angle error or the relative pose error is greater than the corresponding threshold, the reliability label of the VO processing device when processing that image is determined to be unreliable. If both the mounting angle error and the relative pose error are less than or equal to their respective thresholds, the reliability label of the VO processing device when processing that image is determined to be reliable. Furthermore, the pre-set model is trained using the internal variables of the VO processing device when processing each image and the reliability label for each image, thus obtaining a VO reliability assessment model with VO reliability assessment capabilities. Finally, by distributing the trained VO reliability assessment model to terminal devices, the terminal devices can use this model to assess the reliability of VOs, improving the accuracy of VO reliability assessment.

[0040] In the embodiments of this disclosure, when training the VO reliability assessment model, the dependence of downstream modules such as PDR on the installation angle is considered. The installation angle error and relative pose error are used as the basis when calculating the tag, which improves the accuracy of reliability tag determination. Therefore, the VO reliability assessment model is trained based on the reliability tag and the internal variables of the VO processing device in the process of processing image sequences, so that the VO reliability assessment model can accurately assess the reliability of VO, thereby improving the accuracy of VO reliability assessment.

[0041] To better understand the technical solutions of the embodiments of this disclosure, the solutions of the embodiments of this disclosure will be described below in conjunction with exemplary embodiments.

[0042] Example, Figure 2This is a flowchart illustrating a training method for a VO reliability assessment model provided in this disclosure. The method can be executored by a computer device, such as a server, laptop, desktop computer, distributed computing node, or other device with computing and processing capabilities. Figure 2 As shown, in some embodiments of this disclosure, the training method for the VO reliability assessment model may include steps 201-205.

[0043] Step 201: Obtain the image sequence and the actual trajectory and pose of the acquisition device when acquiring the image sequence.

[0044] The acquisition device referred to in the embodiments of this disclosure can be understood as a device with image acquisition capabilities, such as a camera or webcam.

[0045] An image sequence can be understood as an image sequence obtained by arranging multiple consecutive images acquired in the order they were captured.

[0046] The true trajectory and true pose can be understood as the actual trajectory and true pose of the acquisition device when acquiring image sequences. Specifically, each trajectory point in the true trajectory corresponds one-to-one with an image in the image sequence, meaning each trajectory point in the true trajectory corresponds to one image in the image sequence, and each image corresponds to a true pose.

[0047] In some embodiments of this disclosure, the image sequence acquired by the acquisition device, as well as the actual trajectory and pose of the acquisition device during the acquisition of the image sequence, can be obtained from a preset data source. The data source can be an external storage device of the computer device, such as a portable hard drive, a database, an application server, etc., but is not limited to the storage devices listed herein; it can also be an internal storage device of the computer device, such as a hard drive.

[0048] Step 202: Input the image sequence into the VO processing device for processing, and obtain the internal variables of the VO processing device during the image sequence processing process, as well as the estimated trajectory and estimated pose of the acquisition device.

[0049] In this embodiment, the VO processing device can estimate the motion of the acquisition device based on the image sequence acquired by the acquisition device. The VO processing device may include a front end and a back end. The front end processes the input images using visual odometry (VO) to estimate the pose of the acquisition device (i.e., estimated pose), thus completing pose tracking. The back end uses a preset optimization algorithm to optimize the pose calculated by the front end to obtain a globally consistent trajectory (i.e., estimated trajectory). In this embodiment, the trajectory points on the estimated trajectory also correspond one-to-one with the images in the image sequence, and each image corresponds to one estimated pose.

[0050] In this embodiment of the disclosure, the internal variables of the VO processing device during the processing of the image sequence include the internal variables of the VO processing device when processing each image. In practice, when the VO processing device processes the images in the image sequence, the front end of the VO processing device generates feature variables related to front-end processing, the back end generates feature variables related to back-end optimization, and feature variables related to trajectory. The internal variables referred to in this embodiment of the disclosure may include at least one of the above-mentioned feature variables.

[0051] In this embodiment of the disclosure, the feature variables related to front-end processing may include at least one of the following variables: the original value of the covariance between pose state variables and its recent statistics, and whether the pose of the acquisition device was successfully estimated.

[0052] The raw covariance values ​​between pose state variables refer to the covariance variables calculated during the front-end pose calculation process. These variables have seven dimensions: three representing the position of the acquisition device and four representing its attitude. The recent statistics of the raw covariance values ​​between pose state variables refer to the median calculated from the median of the raw covariance values ​​within a preset time period, or the quartiles calculated from the quartiles. Specifically, the method for determining the recent statistics can be set as needed and is not limited here. Whether the pose of the acquisition device was successfully estimated needs to be determined based on the front-end processing results; success and failure are represented by different numerical values.

[0053] Feature variables related to backend optimization can include at least one of the following: original average residual, average residual, scale change before and after optimization, recent average scale change, number of frames optimized within the sliding window, number of optimized points, and number of successfully tracked 3D points. Here, the original average residual refers to the average value of the reprojection residuals during the backend nonlinear optimization process. The average residual refers to the average value of the reprojection residuals after adding a Robin kernel function during the backend nonlinear optimization process. Scale change before and after optimization refers to the change in distance between adjacent images before and after backend nonlinear optimization, indicating the rate of change in distance between images before and after optimization. Recent average scale change refers to caching the "scale change before and after optimization" over a period of time and calculating its average value. The number of frames optimized within the sliding window refers to the number of images involved in optimization during the backend nonlinear optimization process. The number of optimized points refers to the total number of feature points observed by the backend. The number of successfully tracked 3D points refers to the number of feature points that have been successfully tracked and triangulated, reflecting the frontend tracking effect.

[0054] Trajectory-related feature variables include at least one of distance ratio and direction change. The distance ratio refers to the distance l1 between the average of the acquisition locations of a preset number of past images and the acquisition location of the current image, and the distance l2 between the two locations farthest from the origin (which can also be understood as the acquisition starting point) and the two locations closest to the origin in the preset number of past images. The distance ratio is l1 / l2.

[0055] Step 203: Based on the actual trajectory and pose of the acquisition device when acquiring the image sequence and the estimated trajectory and pose estimated by the VO processing device, determine the installation angle error and relative pose error of the VO processing device when processing each image.

[0056] In the embodiments of this disclosure, there can be various methods for determining the mounting angle error and relative pose error of the VO processing device when processing each image. For example, Figure 3 This is a flowchart illustrating a method for determining the installation angle error of a VO processing device when processing images. (For example...) Figure 3 As shown, in one feasible implementation, the installation angle error of the VO processing device when processing each image can be determined by the method in steps S11-S13.

[0057] In S11, based on the actual trajectory and pose of the acquisition device when acquiring each image, the first installation angle of the acquisition device when acquiring each image is determined.

[0058] For example, in one implementation, for any image in the image sequence (hereinafter referred to as the current image), the direction from the current trajectory point to the other trajectory point in the real trajectory (hereinafter referred to as the current trajectory point) can be determined as the current trajectory point's direction of travel, based on the position of the trajectory point of the current image captured in the real trajectory (hereinafter referred to as the current trajectory point) and the position of another trajectory point in the real trajectory located after the current trajectory point (e.g., a trajectory point located after a preset time length). Alternatively, the direction from another trajectory point in the real trajectory located before the current trajectory point to the current trajectory point can be determined as the current trajectory point's direction of travel. Further, the orientation of the acquisition device at the current trajectory point is obtained based on the device's pose at the current trajectory point. Then, the angle between the acquisition device's direction of travel at the current trajectory point and its orientation is determined as the first mounting angle of the acquisition device when capturing the current image. The method for determining the orientation based on the pose can be found in related technologies and will not be detailed here.

[0059] In S12, based on the estimated trajectory and estimated pose of the acquisition device when acquiring each image, the second installation angle of the acquisition device when acquiring each image is determined.

[0060] The method for determining the second mounting angle is similar to that for determining the first mounting angle, and will not be repeated here.

[0061] In S13, for each image, the deviation between the first mounting angle and the second mounting angle when acquiring the image is determined as the mounting angle error of the VO processing device when processing the image.

[0062] In one embodiment of this disclosure, the absolute value of the difference between the first mounting angle and the second mounting angle when the acquisition device acquires an image can be determined as the mounting angle error of the VO processing device when processing the image. Alternatively, the first mounting angle can be used as a reference, and the difference between the first mounting angle and the second mounting angle can be used as the mounting angle error of the VO processing device when processing the image. Of course, this is only an illustrative example and not the only limitation.

[0063] It should be noted that, Figure 3 The method shown is only an example and not the only way to determine the mounting angle error. For example, in other implementations, the actual trajectory, actual pose, estimated trajectory, and estimated pose of the acquisition device when acquiring the image can be input into a pre-trained model, and the model can output the mounting angle error of the VO processing device when processing the image.

[0064] Example, Figure 4 This is a flowchart illustrating a method for determining the relative pose error of a VO processing device when processing images, such as... Figure 4 As shown, in one embodiment, the relative pose error of the VO processing device when processing each image can be determined through steps S21-S24.

[0065] In S21, the true trajectory and the estimated trajectory are aligned to obtain the alignment relationship between the trajectory points on the two trajectories, where the aligned trajectory points correspond to the same image in the image sequence.

[0066] Since there is a one-to-one correspondence between the trajectory points on the true trajectory and the estimated trajectory and the images in the image sequence, in one feasible implementation, the trajectories on the true trajectory and the estimated trajectory corresponding to the same image can be identified as aligned trajectory points based on the correspondence between the trajectory points on the true trajectory and the images in the image sequence, thereby achieving alignment between the true trajectory and the estimated trajectory. Alternatively, in another feasible implementation, the first trajectory point on the true trajectory and the first trajectory point on the estimated trajectory can be aligned, and so on, until the alignment of the entire true trajectory and the estimated trajectory is completed.

[0067] In S22, based on the actual pose of the acquisition device at the first and second trajectory points on the real trajectory, the first relative pose of the acquisition device when acquiring the image at the first trajectory point is determined.

[0068] In this embodiment of the disclosure, the first trajectory point and the second trajectory point can be understood as any two trajectory points on the actual trajectory. For ease of understanding, in this embodiment of the disclosure, the first trajectory point can be understood as the first trajectory point on the actual trajectory, and the second trajectory point can be understood as the second trajectory point on the actual trajectory.

[0069] In this embodiment of the disclosure, the absolute value of the difference between the pose of the acquisition device at the second trajectory point and the pose of the acquisition device at the first trajectory point can be used as the first relative pose of the acquisition device for acquiring the image at the first trajectory point.

[0070] In S23, based on the estimated poses of the acquisition device at the third and fourth trajectory points on the estimated trajectory, the second relative pose of the acquisition device when acquiring the image is determined, wherein the third trajectory point is aligned with the first trajectory point, and the fourth trajectory point is aligned with the second trajectory point.

[0071] In this embodiment, the third trajectory point is a trajectory point on the estimated trajectory aligned with the first trajectory point on the true trajectory. The fourth trajectory point is a trajectory point on the estimated trajectory aligned with the second trajectory point on the true trajectory. In one implementation, the absolute value of the difference between the estimated pose of the acquisition device at the fourth trajectory point and the estimated pose of the acquisition device at the third trajectory point can be used as the second relative pose of the image acquired by the acquisition device at the third trajectory point, wherein the image acquired at the third trajectory point and the image acquired at the first trajectory point are the same image.

[0072] In S24, the deviation between the first relative pose and the second relative pose is determined as the relative pose error of the VO processing device when processing the image.

[0073] In one embodiment of this disclosure, the absolute value of the difference between the first relative pose and the second relative pose can be determined as the relative pose error of the VO processing device when processing the image. Alternatively, the first relative pose can be used as a reference, and the difference between the first relative pose and the second relative pose can be used as the relative pose error of the VO processing device when processing the image. Of course, this is only an example and not a unique limitation.

[0074] It should be noted that, Figure 4 The method shown is only one example and not the only method. For example, in other implementations, the actual trajectory, actual pose, estimated trajectory and estimated pose of the acquisition device when acquiring images can be input into a pre-trained model, and the model can output the relative pose error of the VO processing device when processing each image.

[0075] Step 204: Based on the installation angle error and relative pose error of the VO processing device when processing each image, determine the reliability label of the VO processing device when processing each image.

[0076] For example, in one feasible implementation, for each image in the image sequence, the mounting angle error of the VO processing device when processing the image can be compared with a first preset threshold, and the relative pose error of the VO processing device when processing the image can be compared with a second preset threshold. When the mounting angle error is less than the first preset threshold and the relative pose error is less than the second preset threshold, the reliability label of the VO processing device when processing the image is determined to be reliable. If the mounting angle error is greater than or equal to the first preset threshold, or the relative pose error is greater than or equal to the second preset threshold, the reliability label of the VO processing device when processing the image is determined to be unreliable.

[0077] For example, in other feasible implementations, the mounting angle error and relative pose error of the VO processing device when processing an image can be input into a preset recognition model, and the reliability label of the VO processing device when processing the image can be output through the recognition model.

[0078] Of course, the above two methods are only illustrative examples and not the only limitations. In fact, any method that can apply relative pose error and mounting angle error to determine the reliability label of the VO processing device can be used in the embodiments of this disclosure.

[0079] Step 205: Train the VO reliability assessment model based on the internal variables of the VO processing device during the image sequence processing process and the reliability labels of the VO processing device when processing each image.

[0080] The VO reliability assessment model mentioned in this disclosure can be any known model type. For ease of understanding, the VO reliability assessment model can be exemplarily understood as a machine learning gradient boosting model based on a decision tree algorithm, such as LightGBM, but not limited to LightGBM. LightGBM has the advantage of low computational consumption, effectively saving computing power. The use of LightGBM in this disclosure embodiment enables the model trained in this disclosure embodiment to be widely applied to various terminal devices, especially those with lower computing power.

[0081] In practice, the VO processing device generates corresponding internal variables when processing each image. In this embodiment, when training the VO reliability assessment model, the internal variables corresponding to all images in the image sequence can be used, or the internal variables of a subset of consecutive images in the image sequence can be used. Before starting training, the internal variables corresponding to each image are arranged according to the image acquisition order to obtain a variable set. Then, the variable set and the reliability labels of the VO processing device when processing each image are input into the VO reliability assessment model for training.

[0082] By arranging the internal variables corresponding to each image according to the image acquisition order, the resulting variable set can carry temporal information, thereby improving the accuracy of model training.

[0083] In the embodiments disclosed herein, the training method of the VO reliability assessment model is similar to that of related technologies, and will not be described in detail here.

[0084] By acquiring image sequences and the actual trajectory and pose of the acquisition device during image acquisition, and inputting the image sequences into a VO processing device, the internal variables of the VO processing device during image sequence processing, as well as the estimated trajectory and pose of the acquisition device, are obtained. The mounting angle error and relative pose error of the VO processing device during image processing are determined using the actual trajectory and pose, and the estimated trajectory and pose. Based on these errors, a reliability label is determined for each image. Thus, a VO reliability assessment model is trained using the internal variables of the VO processing device during image sequence processing and the reliability labels for each image. This embodiment of the present disclosure considers the dependence of downstream modules such as PDR on the mounting angle when training the VO reliability assessment model. The label calculation is based on the mounting angle error and relative pose error, improving the accuracy of the reliability labels. Therefore, training the VO reliability assessment model based on the reliability labels and the internal variables of the VO processing device during image sequence processing improves the accuracy of the VO reliability assessment model, thereby improving the accuracy of the VO reliability assessment. The VO reliability assessment model provided in this disclosure can assess the reliability of VO in real time, providing a reliable basis for downstream modules such as PDR.

[0085] For example, in one embodiment provided in this disclosure, before training the VO reliability assessment model based on the internal variables and reliability labels of the VO processing device, the method may further include a step of standardizing and / or data augmenting the internal variables of the VO processing device when processing each image. Standardization includes, but is not limited to, 0-1 standardization and Gaussian standardization. Data augmentation includes, but is not limited to, adding Gaussian random noise. Standardizing the internal variables facilitates their processing, while data augmentation increases the randomness of the samples, thereby improving the accuracy of model training.

[0086] For example, in some embodiments of this disclosure, after determining the mounting angle error and relative pose error of the VO processing device when processing each image based on the true trajectory and true pose, the estimated trajectory and estimated pose, a data filtering step may be included. For instance, in one data filtering method, for the mounting angle error and relative pose error of the VO processing device when processing each image, the mounting angle error can be compared with a third preset threshold, and the relative pose error can be compared with a fourth preset threshold. When the mounting angle error is less than the third preset threshold and the relative pose error is less than the fourth preset threshold, the step of determining the reliability label of the VO processing device when processing the image based on the mounting angle error and relative pose error can be further performed. If the mounting angle error is greater than or equal to the third preset threshold, or the relative pose error is greater than or equal to the fourth preset threshold, then it is determined that the internal variables of the image processed by the VO processing device at this time belong to an abnormal sample, and the internal variables, estimated trajectory, and estimated pose at this time are deleted. Furthermore, the original image sequence can be reprocessed based on the VO processing device to obtain the internal variables of the VO processing device in the reprocessing process, or the current image sequence can be discarded, and a new image sequence can be obtained and processed to obtain new internal variables. Data filtering can remove abnormal samples and retain normal samples, thus ensuring the accuracy of model training.

[0087] Figure 5 This is a flowchart of a VO reliability assessment method provided in an embodiment of this disclosure. This method can be executed by a terminal device (e.g., a mobile phone, tablet, wearable device, or other device with navigation capabilities, but not limited to the devices listed here). The terminal device can use the VO reliability assessment model trained in the above embodiments to assess the reliability of the VO processing device, and perform positioning calculations and navigation based on the assessment results and the PDR algorithm. Figure 5 As shown, the evaluation method includes:

[0088] Step 501: Obtain the first image to be processed.

[0089] Step 502: Input the first image into the VO processing device and obtain the first internal variable of the VO processing device during the process of processing the first image.

[0090] In this embodiment, the parameters and types included in the internal variables are similar to... Figure 2 The internal variables mentioned in the embodiments are consistent and will not be repeated here.

[0091] Step 503: Input the first internal variable and the second internal variable of the VO processing device in the process of processing multiple second images into the pre-trained VO reliability evaluation model to obtain the VO reliability evaluation result of the VO processing device in the first image.

[0092] The pre-trained VO reliability assessment model can be understood as the VO reliability assessment model trained based on the model training method in the aforementioned embodiments.

[0093] Here, multiple second images refer to multiple images captured before the first image.

[0094] Example, Figure 6 This is a schematic diagram of a VO reliability assessment scenario, such as... Figure 6 As shown, the acquisition device inputs the real-time acquired image sequence into the VO processing device for processing. For each received image, the internal variables of the VO processing device during image processing are transmitted to the inference engine. The inference engine contains the VO reliability evaluation model trained according to the embodiments of this disclosure. The inference engine inputs the internal variables of the VO processing device during image processing and the internal variables of the VO processing device during processing of multiple images input before processing this image (pre-stored in the cache) into the VO reliability evaluation model to obtain the VO reliability evaluation result of the VO processing device when processing the current image, and sends the VO reliability evaluation result to the downstream PDR for pose calculation. At the same time, the VO reliability evaluation result is fed back to the VO processing device so that the VO processing device can self-adjust according to the VO reliability evaluation result to improve the reliability of the VO processing device.

[0095] The beneficial effects of the embodiments disclosed herein and Figure 2 The implementation examples are similar and will not be repeated here.

[0096] Figure 7 This is a schematic diagram of the structure of a training device for a VO reliability assessment model provided in this embodiment. This device can be understood as the computer device or a functional module within the computer device described in the above embodiments. Figure 7 As shown, the training device 70 provided in this embodiment includes:

[0097] The first acquisition module 71 is used to acquire the image sequence and the real trajectory and real pose of the acquisition device when acquiring the image sequence;

[0098] The second acquisition module 72 is used to input the image sequence into the VO processing device for processing, and to acquire the internal variables of the VO processing device during the processing of the image sequence, as well as the estimated trajectory and estimated pose of the acquisition device.

[0099] The first determining module 73 is used to determine the installation angle error and relative pose error of the VO processing device when processing each image based on the real trajectory and the real pose, the estimated trajectory and the estimated pose;

[0100] The second determining module 74 is used to determine the reliability label of the VO processing device when processing each image based on the installation angle error and relative pose error of the VO processing device when processing each image.

[0101] Training module 75 is used to train a VO reliability evaluation model based on the internal variables of the VO processing device during the processing of the image sequence and the reliability labels of the VO processing device when processing each image.

[0102] In one implementation, the first determining module 73 is configured to:

[0103] Based on the actual trajectory and pose of the acquisition device when acquiring each image, the first installation angle of the acquisition device when acquiring each image is determined;

[0104] Based on the estimated trajectory and estimated pose of the acquisition device when acquiring each image, the second installation angle of the acquisition device when acquiring each image is determined;

[0105] For each image, the deviation between the first mounting angle and the second mounting angle when the image is acquired is determined as the mounting angle error of the VO processing device when processing the image.

[0106] In one implementation, the first determining module 73 can also be used for:

[0107] Alignment processing is performed on the actual trajectory and the estimated trajectory to obtain the alignment relationship between the trajectory points on the two trajectories, wherein the trajectory points that are aligned with each other correspond to the same image in the image sequence;

[0108] Based on the true pose of the acquisition device at the first and second trajectory points on the real trajectory, the first relative pose of the acquisition device when acquiring the image at the first trajectory point is determined.

[0109] Based on the estimated pose of the acquisition device at the third and fourth trajectory points of the estimated trajectory, the second relative pose of the acquisition device when acquiring the image is determined, wherein the third trajectory point is aligned with the first trajectory point, and the fourth trajectory point is aligned with the second trajectory point;

[0110] The deviation between the first relative pose and the second relative pose is determined as the relative pose error of the VO processing device when processing the image.

[0111] In one implementation, the second determining module 74 is configured to:

[0112] For each image, the installation angle error of the VO processing device when processing the image is compared with a first preset threshold, and the relative pose error of the VO processing device when processing the image is compared with a second preset threshold.

[0113] In response to the installation angle error being less than a first preset threshold and the relative pose error being less than a second preset threshold, the reliability label of the VO processing device when processing the image is determined to be reliable.

[0114] In response to the installation angle error being greater than or equal to a first preset threshold, or the relative pose error being greater than or equal to a second preset threshold, the reliability label of the VO processing device when processing the image is determined to be unreliable.

[0115] In one embodiment, the training device 70 may further include:

[0116] The internal variables of the VO processing device are standardized and / or data augmented when processing each image.

[0117] In one embodiment, the training device 70 may further include a second processing module for:

[0118] For the mounting angle error and relative pose error of the VO processing device when processing each image, the mounting angle error is compared with a third preset threshold, and the relative pose error is compared with a fourth preset threshold.

[0119] In response to the mounting angle error being less than the third preset threshold and the relative pose error being less than the fourth preset threshold, the step of determining the reliability label of the VO processing device when processing each image based on the mounting angle error and relative pose error of the VO processing device when processing each image is executed.

[0120] In response to the installation angle error being greater than or equal to the third preset threshold, or the relative pose error being greater than or equal to the fourth preset threshold, the internal variables of the VO processing device when processing the image, as well as the estimated trajectory and estimated pose of the acquisition device when acquiring the image, are deleted.

[0121] In one implementation, training module 75 is used for:

[0122] According to the acquisition order of each image in the image sequence, the internal variables of the VO processing device during the processing of each image are arranged to obtain a variable group;

[0123] A VO reliability assessment model is trained based on the variable set and the reliability labels of the VO processing device when processing each image.

[0124] The training apparatus provided in this embodiment is capable of performing the above-described... Figures 2-4 The methods in any of the method embodiments are similar in execution and beneficial effects, and will not be described again here.

[0125] Figure 8 This is a schematic diagram of the structure of a VO reliability assessment device provided in an embodiment of this disclosure. The device can be understood as... Figure 5 The terminal device or some functional modules within the terminal device in the embodiments. For example... Figure 8 As shown, the VO reliability assessment device 80 includes:

[0126] The first acquisition module 81 is used to acquire the first image to be processed;

[0127] The second acquisition module 82 is used to input the first image into the VO processing device and acquire the first internal variable of the VO processing device during the process of processing the first image.

[0128] The reliability assessment module 83 is used to input the first internal variable and the second internal variable of the VO processing device in the process of processing multiple second images into a pre-trained VO reliability assessment model to obtain the VO reliability assessment result of the VO processing device when processing the first image; wherein, the second image refers to the image collected before the first image.

[0129] The pre-trained VO reliability evaluation model can be understood as: Figure 7 The VO reliability evaluation model trained in the example.

[0130] The VO reliability assessment device provided in this disclosure is capable of performing... Figure 5 The method described in the embodiments is similar in its execution and beneficial effects, and will not be repeated here.

[0131] This disclosure also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, it can achieve the above-described functionality. Figures 2-4 The method in any of the method embodiments.

[0132] This disclosure also provides a terminal device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, it can perform the functions described above. Figure 5 The method in the method embodiment.

[0133] Example, Figure 9 This is a schematic diagram of the structure of a terminal device according to an embodiment of this disclosure. See below for details. Figure 9 The diagram illustrates a structural schematic suitable for implementing the terminal device 1400 in the embodiments of this disclosure. The terminal device 1400 in the embodiments of this disclosure may include, but is not limited to, devices with data processing and computing capabilities such as mobile phones, tablets, and wearable devices. Figure 9 The terminal device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0134] like Figure 9 As shown, the terminal device 1400 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage device 1408 into a random access memory (RAM) 1403. The RAM 1403 also stores various programs and data required for the operation of the terminal device 1400. The processing unit 1401, ROM 1402, and RAM 1403 are interconnected via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0135] Typically, the following devices can be connected to I / O interface 1405: input devices 1406 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1407 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1408 including, for example, magnetic tape, hard disk, etc.; and communication devices 1409. Communication device 1409 allows terminal device 1400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 A terminal device 1400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0136] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1409, or installed from storage device 1408, or installed from ROM 1402. When the computer program is executed by processing device 1401, it performs the functions defined in the methods of embodiments of this disclosure.

[0137] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0138] The aforementioned computer-readable medium may be included in the aforementioned terminal device; or it may exist independently and not assembled into the terminal device.

[0139] The aforementioned computer-readable medium carries one or more programs, which, when executed by a processing device, cause the processing device to: acquire a first image to be processed; input the first image into a VO processing device, acquire a first internal variable of the VO processing device during the processing of the first image; input the first internal variable and a second internal variable of the VO processing device during the processing of multiple second images into a pre-trained VO reliability assessment model, and obtain a VO reliability assessment result of the VO processing device when processing the first image; wherein, the second image refers to an image acquired before the first image.

[0140] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0142] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0143] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0144] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0145] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can perform the above-described functions. Figures 2-5 The methods in any of the embodiments are similar in execution and beneficial effects, and will not be described again here.

[0146] This disclosure also provides a computer program product, which is stored in a storage medium. When the program product is run, it can achieve... Figures 2-5 The methods in any of the embodiments are similar in execution and beneficial effects, and will not be described again here.

[0147] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0148] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of training a VO reliability assessment model, wherein, include: Acquire the image sequence and the actual trajectory and pose of the acquisition device when acquiring the image sequence; The image sequence is input into the VO processing device for processing. The internal variables of the VO processing device during the processing of the image sequence, as well as the estimated trajectory and estimated pose of the acquisition device, are obtained. The internal variables include the internal variables of the VO processing device when processing each image. Based on the actual trajectory and the actual pose, the estimated trajectory and the estimated pose, the installation angle error and relative pose error of the VO processing device when processing each image are determined; Based on the installation angle error and relative pose error of the VO processing device when processing each image, the reliability label of the VO processing device when processing each image is determined. A VO reliability assessment model is trained based on the internal variables of the VO processing device during the processing of the image sequence and the reliability labels of the VO processing device when processing each image.

2. The method of claim 1, wherein, The determination of the mounting angle error and relative pose error of the VO processing device when processing each image, based on the true trajectory and the true pose, the estimated trajectory and the estimated pose, includes: Based on the actual trajectory and pose of the acquisition device when acquiring each image, the first installation angle of the acquisition device when acquiring each image is determined; Based on the estimated trajectory and estimated pose of the acquisition device when acquiring each image, the second installation angle of the acquisition device when acquiring each image is determined; For each image, the deviation between the first mounting angle and the second mounting angle when the image is acquired is determined as the mounting angle error of the VO processing device when processing the image.

3. The method of claim 1, wherein, The determination of the mounting angle error and relative pose error of the VO processing device when processing each image, based on the true trajectory and the true pose, the estimated trajectory and the estimated pose, includes: Alignment processing is performed on the actual trajectory and the estimated trajectory to obtain the alignment relationship between the trajectory points on the two trajectories, wherein the trajectory points that are aligned with each other correspond to the same image in the image sequence; Based on the true pose of the acquisition device at the first and second trajectory points on the real trajectory, the first relative pose of the acquisition device when acquiring the image at the first trajectory point is determined. Based on the estimated pose of the acquisition device at the third and fourth trajectory points of the estimated trajectory, the second relative pose of the acquisition device when acquiring the image is determined, wherein the third trajectory point is aligned with the first trajectory point, and the fourth trajectory point is aligned with the second trajectory point; The deviation between the first relative pose and the second relative pose is determined as the relative pose error of the VO processing device when processing the image.

4. The method of claim 1, wherein, The determination of the reliability label of the VO processing device when processing each image based on the installation angle error and relative pose error of the VO processing device includes: For each image, the installation angle error of the VO processing device when processing the image is compared with a first preset threshold, and the relative pose error of the VO processing device when processing the image is compared with a second preset threshold. In response to the installation angle error being less than a first preset threshold and the relative pose error being less than a second preset threshold, the reliability label of the VO processing device when processing the image is determined to be reliable. In response to the installation angle error being greater than or equal to a first preset threshold, or the relative pose error being greater than or equal to a second preset threshold, the reliability label of the VO processing device when processing the image is determined to be unreliable.

5. The method according to any one of claims 1-4, wherein, Before training the VO reliability assessment model based on the internal variables of the VO processing device during the processing of the image sequence and the reliability labels of the VO processing device when processing each image, the method further includes: The internal variables of the VO processing device are standardized and / or data augmented when processing each image.

6. The method according to claim 1, wherein, After determining the mounting angle error and relative pose error of the VO processing device when processing each image based on the true trajectory and the true pose, the estimated trajectory and the estimated pose, the method further includes: For the mounting angle error and relative pose error of the VO processing device when processing each image, the mounting angle error is compared with a third preset threshold, and the relative pose error is compared with a fourth preset threshold. In response to the mounting angle error being less than the third preset threshold and the relative pose error being less than the fourth preset threshold, the step of determining the reliability label of the VO processing device when processing each image based on the mounting angle error and relative pose error of the VO processing device when processing each image is executed. In response to the installation angle error being greater than or equal to the third preset threshold, or the relative pose error being greater than or equal to the fourth preset threshold, the internal variables of the VO processing device when processing the image, as well as the estimated trajectory and estimated pose of the acquisition device when acquiring the image, are deleted.

7. The method according to claim 1, wherein, The training of the VO reliability assessment model based on the internal variables of the VO processing device during the processing of the image sequence and the reliability labels of the VO processing device when processing each image includes: According to the acquisition order of each image in the image sequence, the internal variables of the VO processing device during the processing of each image are arranged to obtain a variable group; A VO reliability assessment model is trained based on the variable set and the reliability labels of the VO processing device when processing each image.

8. A VO reliability assessment method, wherein, include: Get the first image to be processed; The first image is input into the VO processing device, and the first internal variable of the VO processing device during the process of processing the first image is obtained. The first internal variable and the second internal variable of the VO processing device during the processing of multiple second images are input into the VO reliability evaluation model trained by the method as described in any one of claims 1-7 to obtain the VO reliability evaluation result of the VO processing device when processing the first image; wherein, the second image refers to the image acquired before the first image.

9. A training device for a VO reliability assessment model, wherein, include: The first acquisition module is used to acquire the image sequence and the real trajectory and real pose of the acquisition device when acquiring the image sequence; The second acquisition module is used to input the image sequence into the VO processing device for processing, and to acquire the internal variables of the VO processing device during the processing of the image sequence, as well as the estimated trajectory and estimated pose of the acquisition device. The internal variables include the internal variables of the VO processing device when processing each image. The first determining module is used to determine the installation angle error and relative pose error of the VO processing device when processing each image based on the real trajectory and the real pose, the estimated trajectory and the estimated pose; The second determining module is used to determine the reliability label of the VO processing device when processing each image based on the installation angle error and relative pose error of the VO processing device when processing each image. The training module is used to train a VO reliability assessment model based on the internal variables of the VO processing device during the processing of the image sequence and the reliability labels of the VO processing device when processing each image.

10. A VO reliability assessment device, wherein, include: The first acquisition module is used to acquire the first image to be processed. The second acquisition module is used to input the first image into the VO processing device and acquire the first internal variable of the VO processing device during the process of processing the first image. The reliability assessment module is used to input the first internal variable and the second internal variable of the VO processing device during the processing of multiple second images into the VO reliability assessment model trained by the method as described in any one of claims 1-7, so as to obtain the VO reliability assessment result of the VO processing device when processing the first image; wherein, the second image refers to the image collected before the first image.

11. A computer device, wherein, It includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 1-7.

12. A terminal device, wherein, It includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method as described in claim 8.

13. A computer program product, wherein, The program product is stored in a storage medium, and when the program product is executed by a processor, it implements the method as described in any one of claims 1-8.

Images