Map construction method and device, computer equipment and storage medium

Abstract

The invention relates to a map construction method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring multiple frames of images shot in a mapping process; calculating the motion amplitude of a camera shooting the multiple frames of images, and calculating the image luminosity change of the multiple frames of images; according to the feature characteristics of adjacent images in the multiple frames of images, the motion amplitude and the image luminosity change, a matched tracking method is selected for tracking, a tracking result is obtained, and the tracking method comprises a feature tracking method, a direct tracking method and an IMU tracking method; and performing map construction according to the tracking result to obtain map elements, and performing loopback according to the feature points of the multiple frames of images to obtain a target map. By adopting the method, when a complex scene is encountered, it can be ensured that mapping cannot be interrupted.

Description

Technical Field

[0001] The present disclosure relates to the technical field of map data processing, and in particular to a map construction method, device, computer equipment, and storage medium. Background Art

[0002] With the continuous development of SLAM technology, simultaneous localization and mapping (SLAM) can use its own specific sensors to determine its own position and build a map of the surrounding environment, while visual SLAM (VSLAM) uses cameras as the main sensor, and its application scenarios are becoming more and more extensive.

[0003] During the mapping process, VSLAM often encounters complex scenes, such as weak textures, repeated textures, and low-light scenes. At this time, the number of feature points in the scene will drop sharply and the quality will deteriorate. Therefore, visual SLAM often loses track because it is difficult to detect and track enough feature points, resulting in SLAM system tracking failure and mapping interruption. Summary of the invention

[0004] Based on this, it is necessary to provide a map construction method, device, computer equipment, and storage medium to address the above technical problems.

[0005] In a first aspect, the present disclosure provides a map construction method. The method comprises:

[0006] Obtain multiple frames of images captured during the mapping process;

[0007] Calculating the motion amplitude of the camera shooting the multiple frames of images, and calculating the image brightness change of the multiple frames of images;

[0008] According to the characteristic characteristics of adjacent images in the multiple frames of images, the motion amplitude and the image brightness change, a matching tracking method is selected for tracking to obtain a tracking result, wherein the tracking method includes: a feature tracking method, a direct tracking method and an IMU tracking method;

[0009] A map is constructed according to the tracking results to obtain map elements, and a loop is performed according to the feature points of the multiple frames of images to obtain a target map.

[0010] In one embodiment, the characteristic characteristics include: characteristic quantity and characteristic quality; the selecting a matching tracking method for tracking according to the characteristic characteristics of adjacent images in the multiple frames of images, the motion amplitude and the image brightness change to obtain a tracking result includes:

[0011] In response to satisfying a first selection condition of the feature tracking method, selecting the feature tracking method for tracking to obtain a tracking result, wherein the first selection condition includes: the number of features is greater than a preset feature number threshold, and the quality of the features is greater than a preset quality index;

[0012] In response to satisfying the second selection condition of the direct tracking method, selecting the direct tracking method for tracking to obtain a tracking result; the second selection condition includes: the first selection condition is not satisfied, and the motion amplitude is less than a preset amplitude threshold, and the image luminance change is less than a preset change threshold;

[0013] In response to the second selection condition not being met, an IMU tracking method is selected for tracking to obtain a tracking result.

[0014] In one embodiment, the calculating the motion amplitude of the camera shooting the multiple frames of images includes:

[0015] For each frame of the multiple frames of images, determining a feature point in each frame of the image;

[0016] Perform feature matching on the feature points in each frame of the image to obtain the matching results;

[0017] At least calculating the first pose of the camera in each frame of the image based on the matching results and using epipolar geometry;

[0018] Determine the camera's motion amplitude based on the first pose of the camera in each frame of the image;

[0019] or,

[0020] Obtaining the posture data recorded by the IMU in the camera;

[0021] The posture data is integrated to obtain a second posture of the camera, and the movement amplitude of the camera is determined according to the second posture.

[0022] In one embodiment, in response to satisfying the first selection condition of the feature tracking method, selecting the feature tracking method for tracking to obtain the tracking result includes:

[0023] In response to satisfying the first selection condition of the feature tracking method, acquiring the posture data recorded by the IMU in the camera;

[0024] Time-aligning the timestamp of the posture data recorded by the IMU with the timestamp of the multiple frames of images;

[0025] Based on the time-aligned posture data and the first posture of the camera, posture estimation is performed to obtain a tracking result, wherein the first posture is calculated based on multiple frames of images.

[0026] In one embodiment, the posture data includes: angular velocity and acceleration; in response to the second selection condition not being met, selecting the IMU tracking method for tracking to obtain a tracking result includes:

[0027] In response to the second selection condition not being met, integrating the angular velocity to obtain the attitude orientation of the camera, and integrating the acceleration to obtain the velocity of the camera;

[0028] Calculating a second posture of the camera based on the posture orientation of the camera and the speed of the camera;

[0029] Perform posture estimation according to the second posture of the camera to obtain a tracking result.

[0030] In one embodiment, the method further comprises:

[0031] In response to satisfying a zero-speed calibration condition, obtaining zero-speed attitude data obtained by the IMU at zero speed, wherein the zero-speed calibration condition includes: a first difference between the IMU acceleration and the gravity acceleration in the IMU is less than a preset difference threshold, a first variance between the IMU acceleration and the gravity acceleration is less than a preset variance threshold, and a variance of the de-zeroed angular velocity is less than the variance threshold;

[0032] Calculating a state error according to the zero-speed posture data and the tracking result;

[0033] Recalculate the angular velocity zero bias in the IMU, and calibrate the roll angle and pitch angle using the projection of the gravitational acceleration;

[0034] The attitude data recorded by the IMU is calibrated using Kalman filtering and the state error.

[0035] In one embodiment, the multiple frames of images are panoramic images, and the method further includes:

[0036] Obtain multiple target maps from multiple terminal devices;

[0037] Determine an initial position in each target map, and reposition according to the panoramic image and the initial position to determine the position of each target map;

[0038] Based on the location of each target map, the multiple target maps are fused.

[0039] In a second aspect, the present disclosure further provides a map construction device. The device comprises:

[0040] An image acquisition module is used to acquire multiple frames of images captured during the mapping process;

[0041] A calculation module, used to calculate the motion amplitude of the camera used to shoot the multiple frames of images, and to calculate the image brightness change of the multiple frames of images;

[0042] A tracking module, used for selecting a matching tracking method for tracking according to the characteristic characteristics of adjacent images in the multiple frames of images, the motion amplitude and the image brightness change, to obtain a tracking result, wherein the tracking method includes: a feature tracking method, a direct tracking method and an IMU tracking method;

[0043] The map construction module is used to construct a map according to the tracking results to obtain map elements, and to perform loop closure according to the feature points of the multiple frames of images to obtain a target map.

[0044] In a third aspect, the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any of the above method embodiments are implemented.

[0045] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0046] In a fifth aspect, the present disclosure further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0047] In the above embodiments, since the movement amplitude of the camera will affect the calculation of the IMU, the image luminance change can affect the image quality and the accuracy of subsequent processing (such as feature extraction, object recognition, etc.), and the characteristic characteristics of the image characterize the number of features and the quality of the features. The above three types of information have a greater impact on visual SLAM. Therefore, according to the characteristic characteristics of adjacent images in the multi-frame images, the movement amplitude and the image luminance change, a matching tracking method can be selected for tracking to obtain tracking results, thereby ensuring that the selected tracking method can achieve better results and the accuracy of the tracking results. A map is constructed according to the tracking results to obtain map elements, and a loop is performed based on the feature points of the multi-frame images to obtain a target map, thereby ensuring the accuracy of the target map construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A diagram of an application environment of a map construction method in an embodiment;

[0050] Figure 2 A schematic diagram of a process of a map construction method in one embodiment;

[0051] Figure 3 This is a schematic diagram of the process of step S206 in one embodiment;

[0052] Figure 4 Schematic diagram of the process of step S204 in one embodiment;

[0053] Figure 5 Schematic diagram of the process of step S302 in one embodiment;

[0054] Figure 6 Schematic diagram of the process of step S306 in one embodiment;

[0055] Figure 7 Schematic diagram of a zero-speed calibration process in one embodiment;

[0056] Figure 8 A schematic diagram of a process of target map fusion step in an embodiment;

[0057] Fig. 9 It is a structural block diagram of a map construction device in one embodiment;

[0058] Fig.10 is an internal structure diagram of a computer device in one embodiment;

[0059] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0061] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0062] In this article, the term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship.

[0063] The present disclosure provides a map construction method which can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 can obtain multiple frames of images taken during the mapping process. The server 104 can calculate the motion amplitude of the camera that takes the multiple frames of images, and calculate the image luminance change of the multiple frames of images. The server 104 selects a matching tracking method for tracking according to the characteristic characteristics of adjacent images in the multiple frames of images, the motion amplitude and the image luminance change, and obtains the tracking result. The server 104 constructs a map according to the tracking results to obtain map elements, and then loops back according to the feature points of the multiple frames of images to obtain the target map. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablet computers, usually the terminal 102 can be a device that can take images and is equipped with an IMU. In some embodiments of the present disclosure, the terminal 102 can be a panoramic camera. The server 104 can be implemented by an independent server or a server cluster consisting of multiple servers. It should be noted that this solution can also be implemented by terminal 102 alone.

[0064] In one embodiment, Figure 2 As shown, a map construction method is provided, which is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0065] S202, obtaining multiple frames of images captured during the mapping process.

[0066] The mapping process usually refers to the process of VSLAM (visual SLAM). Since VSLAM (visual SLAM) needs to use a camera for processing when building a map, visual SLAM mainly relies on the image data captured by the camera to achieve this goal.

[0067] Specifically, during the VSLAM mapping process, a camera can be used to continuously capture images to obtain multiple frames of images. The image can be a panoramic image or a normal image. Usually, in order to better perform loop closure, in some embodiments of the present disclosure, the image can refer to a panoramic image.

[0068] S204, calculating the movement amplitude of the camera used to shoot the multiple frames of images, and calculating the image brightness change of the multiple frames of images.

[0069] Among them, the camera motion amplitude usually refers to the change in the camera's posture. Image photometric change refers to the phenomenon that the brightness (luminosity) of pixels in the image changes due to changes in factors such as lighting conditions, observation angles, and surface characteristics of objects. This change may affect the quality of the image and the accuracy of subsequent processing (such as feature extraction, object recognition, etc.). In VSLAM, photometric changes may affect the accuracy of feature matching and map construction. The camera motion amplitude can be described as the total movement distance and rotation angle of the camera in a specific time.

[0070] Specifically, the calculation of the camera motion amplitude usually involves the displacement and rotation of the camera in three-dimensional space, so the camera's IMU information can be used to calculate the camera's motion amplitude, or the camera's motion amplitude can be calculated based on the feature points in the multiple frames of images taken by the camera. Calculating the image photometric change between multiple frames usually involves analyzing multiple image sequences to evaluate their brightness changes over time. You can use an image processing library (such as OpenCV) to read multiple frames of images, and then convert each frame into a grayscale image. The image photometric change can be evaluated by calculating the absolute difference between adjacent frames, the average photometric change can be calculated to evaluate the image photometric change, or the maximum photometric change can be calculated to evaluate the image photometric change. Relative differences can also be used to evaluate image photometric changes, which can be calculated using the following formula:

[0071] Among them, 11 and 12 are the pixel values ​​at the same position in adjacent frames.

[0072] S206, selecting a matching tracking method for tracking according to the characteristic characteristics of adjacent images in the multiple frames of images, the motion amplitude and the image brightness change to obtain a tracking result, wherein the tracking method includes: a feature tracking method, a direct tracking method and an IMU tracking method.

[0073] Among them, the feature tracking method can be Visual-Inertial Odometry (VIO), which is a technology that combines measurement information from visual sensors (such as cameras) and inertial sensors (such as inertial measurement units IMU) to estimate the position and posture of moving objects (such as robots, drones or smart phones). The direct tracking method uses all or part of the pixel information of the image to perform motion estimation and three-dimensional reconstruction directly through the photometric consistency assumption (that is, the same scene point has the same pixel intensity in different images). Unlike the feature tracking method, the direct tracking method does not rely on the detection and description of feature points, but through direct comparison of pixel intensities. The IMU tracking method can be to directly use IMU information for separate tracking and use IMU information alone to calculate the posture. Feature characteristics may include feature quantity and feature quality in some embodiments of the present disclosure.

[0074] Specifically, due to the characteristic characteristics of multiple frames, the camera's motion amplitude, and the image brightness changes in multiple frames, they will have a certain degree of impact on mapping. In order to ensure the accuracy of the mapping process, the most suitable tracking method can be determined according to the characteristic characteristics of adjacent images, the camera's motion amplitude, and the image brightness changes.

[0075] In some exemplary embodiments, for example, if the feature characteristics are poor and the camera movement amplitude is large, the IMU tracking method can be used. If the feature characteristics are good, the feature tracking method can be used. If the image brightness changes greatly, the direct tracking method can be used.

[0076] S208, constructing a map according to the tracking results to obtain map elements, and performing looping according to the feature points of the multiple frames of images to obtain a target map.

[0077] Among them, loop closure can be a step in VSLAM, which can include: loop detection and loop calibration. Loop closure detection (Loop Closure Detection) is a crucial technology in robot navigation, visual SLAM (simultaneous localization and mapping) and computer vision. Its main purpose is to identify a specific location that the robot or camera has passed, that is, a "loop" formed in the environment. By identifying these loops, the system can effectively avoid positioning drift and improve the accuracy of the map. Loop closure calibration (Loop Closure Correction) is a concept closely related to loop closure detection, which is of great significance in visual SLAM (simultaneous localization and mapping) and other related robot navigation applications. The process of loop calibration uses the identified loop information to adjust and correct the pose estimation and map generated when moving in the environment to improve the accuracy and consistency of the overall system. Map elements in some embodiments of the present disclosure can generally be feature points converted into maps. Map elements can generally be point clouds. Point cloud is a data set consisting of a set of points defined in three-dimensional space, each point usually contains its coordinates (x, y, z) in three-dimensional space, and possible additional information such as color, intensity or normal vector, etc. Point cloud is widely used in computer vision, robotics, geographic information system (GIS), building information modeling (BIM) and other fields.

[0078] Specifically, after obtaining the tracking results, the tracking results will form a point cloud, and the sparse point cloud formed is the basic component of the map (map elements). In addition, it should be noted that because the features extracted from the image are two-dimensional pixel points, not three-dimensional, and the map points are three-dimensional, it is necessary to convert the two-dimensional feature points into three-dimensional spatial points. At this time, it is necessary to triangulate the feature points through the parallax of the same feature points in adjacent images to obtain 3D points. After obtaining the map elements, a basic map can be formed. Then, looping is performed according to the feature points in multiple frames of images to reduce the error of the map.

[0079] In some exemplary embodiments, the multiple-frame images may be panoramic images. The panoramic images have a 360° panoramic viewing angle. Their wide field of view also facilitates the triggering of loop detection. When a loop is detected, the system will perform loop calibration to reduce the cumulative error of the system during long-term mapping.

[0080] In the above-mentioned map construction method, since the camera's motion amplitude will affect the IMU calculation, the image luminance change can affect the image quality and the accuracy of subsequent processing (such as feature extraction, object recognition, etc.), and the characteristic characteristics of the image characterize the number of features and the quality of the features. The above three types of information have a greater impact on visual SLAM. Therefore, according to the characteristic characteristics of adjacent images in the multi-frame images, the motion amplitude and the image luminance change, a matching tracking method can be selected for tracking to obtain tracking results, thereby ensuring that the selected tracking method can achieve better results and the accuracy of the tracking results. Map construction is performed according to the tracking results to obtain map elements, and a loop is performed according to the feature points of the multi-frame images to obtain a target map, thereby ensuring the accuracy of target map construction.

[0081] In one embodiment, Figure 3 As shown, the feature characteristics include: feature quantity and feature quality. Among them, the feature quality can be measured by some indicators, such as contrast, edge strength. In some exemplary embodiments, the higher the contrast of the area around the feature point, the better the quality of the feature point. The contrast can be evaluated by calculating the brightness change of the pixels around the feature point. The edge detection algorithm can also be used to detect whether the feature point is located on the edge or corner point. The higher the edge strength, the better the quality of the feature point. The edge detection algorithm (such as Canny edge detection) can be used for evaluation. In addition, the Harris corner detection algorithm can be used to calculate the response value of the feature point. The higher the response value, the better the quality of the feature point. Or the FAST algorithm can be used to calculate the response of the feature point. The higher the response value, the better the quality of the feature point.

[0082] The step of selecting a matching tracking method for tracking based on the characteristic characteristics of adjacent images in the multiple frames of images, the motion amplitude and the image brightness change to obtain a tracking result includes:

[0083] S302, in response to satisfying the first selection condition of the feature tracking method, selecting the feature tracking method for tracking to obtain a tracking result, wherein the first selection condition includes: the number of features is greater than a preset feature number threshold, and the feature quality is greater than a preset quality index.

[0084] Among them, those skilled in the art can set and adjust the feature quantity threshold and quality index according to actual conditions. In some embodiments of the present disclosure, the feature quantity threshold and quality index are not restricted.

[0085] Specifically, under normal circumstances, the feature tracking method can be selected by default for tracking. In the process of mapping, the number of features and the quality of features are constantly detected. When the number of features is greater than the preset feature number threshold, and the quality of the features is greater than the preset quality index, at this time, since the features meet the requirements, there is no need to switch to other methods, and the feature tracking method can be used for processing to ensure the quality of tracking.

[0086] S304, in response to satisfying the second selection condition of the direct tracking method, selecting the direct tracking method for tracking to obtain a tracking result; the second selection condition includes: the first selection condition is not satisfied, and the motion amplitude is less than a preset amplitude threshold, and the image luminance change is less than a preset change threshold.

[0087] Specifically, when the feature quality is less than the quality index or the feature quantity is less than the feature quantity threshold, it can be determined that the first selection condition is not met at present. When the first selection condition is not met, since the feature quantity and feature quality do not meet the requirements, if the feature tracking law is used, a large error will occur, and other methods can be used for tracking. At this time, other tracking methods need to be selected, so the corresponding tracking method can be selected according to the motion amplitude and image luminance change. When the camera's motion amplitude does not change much, the motion amplitude is less than the preset amplitude threshold, and the image luminance change is less than the preset change threshold, if the IMU tracking method is used at this time, due to the small motion amplitude, the accuracy of the IMU tracking method is not high, so the direct tracking method can be used for tracking.

[0088] S306: In response to the second selection condition not being met, selecting an IMU tracking method for tracking to obtain a tracking result.

[0089] Specifically, the image brightness change may cause the appearance of the same object in different images to be different, affecting the accuracy of classification and recognition. Image brightness change may make it difficult to align images, affecting the final registration effect. Therefore, when the image brightness change is large, the effect of using the direct tracking method for tracking is poor. When the camera moves with a large amplitude, the IMU data changes more obviously. Therefore, the IMU tracking method can be used directly to track and obtain the tracking result.

[0090] In addition, it should be noted that when selecting to use the direct tracking method or the IMU tracking method, if the first selection condition is met, the tracking method will be switched to the feature tracking method. When the IMU tracking method is used, if the second selection condition is met, the tracking method will be switched to the direct tracking method.

[0091] In this embodiment, by meeting different selection conditions and selecting the corresponding tracking method for tracking, the accuracy of the tracking results can be guaranteed. In the direct tracking method, since it does not rely on features but estimates motion through pixel grayscale information, stable tracking can be performed even in an environment with a small number of features or poor quality. Since IMU work does not rely on the external environment, it can still work normally in a complex environment. Therefore, when the second selection condition is not met, the IMU tracking method can be used for processing to ensure the stability of tracking.

[0092] In one embodiment, Figure 4 As shown, the calculation of the motion amplitude of the camera shooting the multiple frames of images includes:

[0093] S402: For each frame of the multiple frames of images, determine feature points in each frame of the image.

[0094] S404, performing feature matching on feature points in each frame of image to obtain matching results.

[0095] S406, calculating the first pose of the camera in each frame of image at least according to the matching result and using epipolar geometry.

[0096] S408, determining the movement amplitude of the camera according to the first position of the camera in each frame of image.

[0097] Among them, epipolar geometry provides a method to estimate the motion (pose) of a camera by analyzing the geometric relationship between two images. Epipolar geometry refers to the geometric relationship between points in an image and their corresponding epipolar points given a pair of cameras (or images) in stereo vision. The epipolar point refers to the position of the corresponding point of a point in one image in the other image. For any position of the camera, there is an epipolar line, and all corresponding points must lie on this line.

[0098] Specifically, for each frame of image, feature points in each frame of image can be extracted, and feature points can include: point features and line features. Each point feature and line feature in the image has a descriptor that identifies the identity, and each descriptor is different. When two images are matched, the descriptors in the two images are compared, and the match is considered successful if the descriptors are the same. After the match is successful, the two descriptors are considered to be the same features in the two frames of image. It should be noted that when matching features, two adjacent frames of image are usually matched. For example, there are four frames of image in total. The feature points in the first frame of image can be matched with the feature points in the second frame of image, and the feature points in the second frame of image can be matched with the feature points in the third frame of image. The feature points in the third frame of image can be matched with the feature points in the fourth frame of image, thereby obtaining multiple matching results between two adjacent frames of image. Then, the first pose of the camera in each frame of image is calculated based on the matching results and using epipolar geometry. Then, the motion amplitude of the camera is determined based on the first pose of the camera in each frame of image.

[0099] In some exemplary embodiments, after the matching results are obtained, the corresponding feature points can be determined according to the matching results. The basic matrix is ​​calculated using the matched feature point pairs. The basic matrix can be estimated by methods such as the Eight-point Algorithm. The basic matrix is ​​converted into an essential matrix according to the intrinsic parameters of the camera to obtain the relative pose information of the camera. The essential matrix is ​​decomposed into a rotation matrix and a translation vector to obtain the pose change of the camera between two frames. Finally, the pose estimation can be further optimized by nonlinear optimization (such as Bundle Adjustment) to ensure the overall geometric consistency.

[0100] S410, acquiring the posture data recorded by the IMU in the camera.

[0101] S412: Integrate the posture data to obtain a second posture of the camera, and determine a movement amplitude of the camera according to the second posture.

[0102] Among them, the posture data may include acceleration data, angular velocity data, etc.

[0103] Specifically, when performing visual SLAM, a camera or shooting device is usually used to capture images, so the attitude data recorded by the IMU in the camera or shooting device can be obtained. The attitude data recorded by the IMU mainly includes acceleration and angular velocity. By integrating the angular velocity, the angle can be obtained, that is, the attitude direction of the camera. By integrating the acceleration, the velocity can be obtained, and then by integrating the velocity, the displacement can be obtained, and then the movement amplitude of the camera can be determined based on the displacement.

[0104] In this embodiment, the camera motion amplitude is calculated in two different ways, and a suitable way can be selected to calculate the camera motion amplitude according to different scenes, thereby ensuring the accuracy of motion amplitude calculation.

[0105] In one embodiment, Figure 5 As shown, in response to satisfying the first selection condition of the feature tracking method, selecting the feature tracking method for tracking to obtain the tracking result includes:

[0106] S502: In response to satisfying the first condition of the feature tracking method, acquiring posture data recorded by the IMU in the camera.

[0107] S504: Time-align the timestamp of the posture data recorded by the IMU with the timestamp of the multiple frames of images.

[0108] S506, performing posture estimation based on the posture data after time alignment and the first posture of the camera to obtain a tracking result, wherein the first posture is calculated based on multiple frames of images.

[0109] Specifically, when the first selection condition is met, the feature tracking method is usually used for processing. Therefore, the posture data recorded by the IMU in the camera can be obtained, and the timestamp of the posture data can be determined. The feature tracking method is usually processed using the posture data in the IMU and the feature points in the image. Therefore, the timestamp of each frame in the multi-frame image can also be determined. Then, each frame in the multi-frame image is time-aligned with the posture data according to the timestamp. Then, the multi-frame image is used to calculate the first pose of the camera. Then, the posture data and the first pose after time alignment are used for posture estimation to obtain the final tracking result. Furthermore, since the posture data of the IMU and the first pose of the camera are calculated in their respective coordinate systems for posture and displacement, they are different. If we unify them into one coordinate system, they are the same. Therefore, we unify the two into one coordinate system, and the difference between the two is zero, but this is an ideal state. Generally, there will be errors, resulting in the difference between the two being not zero. Therefore, an optimization problem should be established to obtain the optimal solution, so as to facilitate subsequent fusion. Therefore, before performing the feature tracking method, the vision can be initialized, and the vision and IMU can be initialized. Visual initialization includes extracting and matching point and line features when there are sufficient features, using epipolar geometry to calculate inter-frame poses, and then constructing sliding window optimization to triangulate 3D points. Visual and IMU initialization includes using image and IMU pre-integration data to construct sliding window optimization of gyroscope zero bias, gravity acceleration, scale and other state quantities to obtain optimized values, which facilitates the fusion of subsequent image features and IMU pose data. After obtaining the optimized value, the pose data and the first pose are fused using the optimized value to obtain the final tracking result.

[0110] In this embodiment, in the feature tracking method, by fusing the first pose obtained by image calculation with the pose data of the IMU, the accuracy of tracking can be guaranteed, thereby improving the accuracy of map construction.

[0111] In one embodiment, the posture data includes: angular velocity and acceleration, such as Figure 6 As shown, in response to the second selection condition not being met, selecting the IMU tracking method for tracking to obtain a tracking result includes:

[0112] S602: In response to the second selection condition not being met, integrating the angular velocity to obtain a posture orientation of the camera, and integrating the acceleration to obtain a velocity of the camera.

[0113] S604: Calculate a second posture of the camera based on the posture direction of the camera and the speed of the camera.

[0114] S606: Perform posture estimation according to the second posture of the camera to obtain a tracking result.

[0115] Specifically, the angular velocity can be integrated to get the angle, that is, the orientation of the camera. The acceleration can be integrated to get the velocity, and the velocity can be integrated to get the displacement. Then, based on the attitude superterm and displacement, the camera PQV (Position-Velocity Quaternion) can be determined. P is Position, V is Velocity, and Q is Quaternion, a representation of attitude. PQV can be the second attitude.

[0116] In this embodiment, the posture data (angular velocity and acceleration) of the IMU can be used to accurately estimate the posture of the camera, thereby ensuring the tracking effect, and the posture data of the IMU will not be affected by environmental factors. When the image quality is poor and feature tracking and direct tracking methods cannot be performed, the use of the IMU posture data can ensure the accuracy of tracking.

[0117] In one embodiment, Figure 7 As shown, the method also includes:

[0118] S702, in response to satisfying the zero-speed calibration condition, obtaining zero-speed attitude data obtained by the IMU at zero speed, wherein the zero-speed calibration condition includes: a first difference between the IMU acceleration and the gravity acceleration in the IMU is less than a preset difference threshold, a first variance between the IMU acceleration and the gravity acceleration is less than a preset variance threshold, and a variance of the de-zeroed angular velocity is less than the variance threshold.

[0119] Among them, zero-speed calibration can usually be a technique for calibration during zero-speed detection. Zero-speed calibration can be a ZUPT (Zero Velocity Update) calibration. Zero-speed detection (Zero Velocity Detection, ZVD) is a technique for processing inertial measurement unit (IMU) data, the main purpose of which is to identify the situation when an object is in a stationary state (i.e., the velocity is zero) under specific conditions. This is an important technology in the fields of navigation, positioning, and attitude estimation, especially in inertial navigation systems. The main purpose of zero-speed detection is to reduce the error caused by IMU drift. In the accelerometer and gyroscope of the IMU, long-term use may cause cumulative errors due to noise and other factors. By detecting that the motion state is zero, the system can reset certain parameters to improve the accuracy of subsequent state estimation. When a moving object is in a stationary state, its velocity should theoretically be zero. Zero-speed detection technology determines whether the object is in a zero-speed state by analyzing the data of the accelerometer and gyroscope. After being in a zero-speed state, zero-speed calibration can be performed. Bias-Free Angular Velocity refers to the true angular velocity value after removing the bias error of the sensor when measuring angular velocity. Bias refers to the non-zero value output by the sensor when there is no rotation. This error will affect the accuracy of the angular velocity. Zero-speed attitude data can usually be the data detected by the IMU in a stationary or motionless state.

[0120] Specifically, the error generated by the IMU integration will accumulate rapidly and become larger, so zero-speed detection will be continuously performed. If the zero-speed calibration condition is met, ZUPT calibration will be performed. The stability and accuracy of the IMU's long-term tracking can be improved through ZUPT calibration, so the first difference between the acceleration and gravity acceleration in the IMU can be calculated. And the first variance between the acceleration and gravity acceleration in the IMU is calculated. The variance of the zero-bias angular velocity can also be calculated. When the first difference is less than the preset difference threshold, and the first variance and the variance of the zero-bias angular velocity are both less than the variance threshold, it can be determined that the zero-speed calibration condition needs to be met, and zero-speed calibration is required at this time. Therefore, the zero-speed attitude data obtained by the IMU at zero speed can be obtained.

[0121] S704: Calculate a state error according to the zero-speed posture data and the tracking result.

[0122] Specifically, it should be noted that the tracking results here may include: tracking results obtained using a feature tracking method or tracking results obtained using an IMU tracking method. Since the tracking results obtained by the above two methods will use the posture data in the IMU, the tracking result can be subtracted from the zero-speed posture data to obtain the state error.

[0123] S706, recalculating the angular velocity zero bias in the IMU, and calibrating the roll angle and the pitch angle using the projection of the gravity acceleration.

[0124] Among them, Roll (roll angle) and Pitch (pitch angle) are two important angles that describe the posture of an object in three-dimensional space. The Roll angle is the angle at which an object rotates around its forward direction (usually the x-axis). It describes the degree of roll of the object. When the right side of the object tilts downward, the Roll angle is positive; when the left side tilts downward, the Roll angle is negative. The Pitch angle is the angle at which an object rotates around its lateral axis (usually the y-axis). It describes the degree of up and down tilt of the object. When the front of the object tilts upward, the Pitch angle is positive; when the front tilts downward, the Pitch angle is negative.

[0125] Specifically, the angular velocity zero bias can be recalculated to estimate the current velocity, and then the projection of gravity acceleration is used to calibrate the roll angle and pitch angle.

[0126] S708, calibrating the posture data recorded by the IMU using Kalman filtering and the state error.

[0127] Specifically, the EKF can be used to update the state error, feed back the estimated error amount, and correct the attitude data of the IMU accelerometer, such as zero bias, position, attitude, velocity, etc., according to the feedback information to achieve ZUPT calibration.

[0128] In this embodiment, by correcting the posture data, the stability and accuracy of IMU long-term tracking can be improved.

[0129] In one embodiment, Figure 8 As shown, the multiple frames of images are panoramic images, and the method further includes:

[0130] S802, obtaining multiple target maps from multiple terminal devices.

[0131] S804, determining an initial position in each target map, and repositioning according to the panoramic image and the initial position to determine the position of each target map.

[0132] S806: Based on the location of each target map, perform map fusion on the multiple target maps.

[0133] Relocalization refers to the process of re-determining the position and posture of a device in a known environment in a mobile device (such as a robot, drone, smartphone, etc.) or a virtual reality (VR) system. Relocalization is usually performed after the device loses tracking or the environment changes to ensure that the system can accurately understand its current state.

[0134] Specifically, in the process of mapping a large scene, it is often necessary to build maps multiple times, and multiple devices build maps at the same time. Therefore, it is necessary to fuse the maps collected by different devices at different times to complete the reconstruction of the entire scene. Therefore, in the process of mapping a large scene, multiple target maps obtained by multiple terminal devices in accordance with the above embodiments can be obtained. It should be noted that a GPS module is provided in the terminal device or camera involved in some embodiments of the present disclosure. Therefore, the position information corresponding to the multiple frames of images captured during the mapping process of the terminal device can be determined. According to the position information of the first frame of the image, the initial position of each target map can be determined. Then, the initial position of each target map and the panoramic image at the initial position are used to reposition within a certain range at the initial position, and the position of the target map in the large scene mapping is quickly determined. Then, according to the position of each target map, the target maps are fused to obtain the final large scene mapping.

[0135] In this embodiment, in large-scene mapping, multiple devices can be used to build maps simultaneously, which can improve the speed of mapping, and the panoramic image and the initial position can be used to reposition the target maps obtained by multiple devices and then fuse them, which can improve the mapping speed while ensuring the accuracy of the final large-scene mapping.

[0136] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0137] Based on the same inventive concept, the embodiment of the present disclosure also provides a map construction device for implementing the map construction method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more map construction device embodiments provided below can refer to the limitations on the map construction method above, and will not be repeated here.

[0138] In one embodiment, Fig. 9As shown, a map construction device 900 is provided, comprising: an image acquisition module 902, a calculation module 904, a tracking module 906 and a map construction module 908, wherein:

[0139] An image acquisition module 902 is used to acquire multiple frames of images captured during the mapping process;

[0140] A calculation module 904 is used to calculate the movement amplitude of the camera used to shoot the multiple frames of images, and calculate the image brightness change of the multiple frames of images;

[0141] A tracking module 906 is used to select a matching tracking method for tracking according to the characteristic characteristics of adjacent images in the multiple frames of images, the motion amplitude and the image brightness change to obtain a tracking result, wherein the tracking method includes: a feature tracking method, a direct tracking method and an IMU tracking method;

[0142] The map construction module 908 is used to construct a map according to the tracking results to obtain map elements, and to perform looping according to the feature points of the multiple frames of images to obtain a target map.

[0143] In one embodiment of the apparatus, the characteristic properties include: characteristic quantity and characteristic quality.

[0144] The tracking module 906 includes:

[0145] A feature tracking module, configured to select a feature tracking method for tracking in response to satisfying a first selection condition of the feature tracking method, and obtain a tracking result, wherein the first selection condition includes: the number of features is greater than a preset feature number threshold, and the quality of the features is greater than a preset quality index;

[0146] A direct tracking module, configured to select the direct tracking method for tracking in response to satisfying a second selection condition of the direct tracking method, and obtain a tracking result; the second selection condition includes: the first selection condition is not satisfied, and the motion amplitude is less than a preset amplitude threshold, and the image brightness change is less than a preset change threshold;

[0147] The IMU tracking module is used to select the IMU tracking method for tracking in response to the second selection condition not being met to obtain a tracking result.

[0148] In one embodiment of the device, the calculation module 904 includes:

[0149] The first camera amplitude calculation module is used to determine the feature points in each frame of the multiple images, perform feature matching on the feature points in each frame to obtain matching results, and calculate the first position of the camera in each frame based on at least the matching results and using epipolar geometry; determine the movement amplitude of the camera based on the first position of the camera in each frame.

[0150] The second camera amplitude calculation module is used to obtain the posture data recorded by the IMU in the camera; integrate the posture data to obtain the second posture of the camera, and determine the movement amplitude of the camera according to the second posture.

[0151] In one embodiment of the device, the feature tracking module is also used to obtain the posture data recorded by the IMU in the camera in response to satisfying the first selection condition of the feature tracking method; time-align the timestamp of the posture data recorded by the IMU with the timestamp of the multiple frames of images; perform posture estimation based on the posture data after time alignment and the first posture of the camera to obtain a tracking result, wherein the first posture is calculated based on the multiple frames of images.

[0152] In one embodiment of the device, the posture data includes: angular velocity and acceleration; the IMU tracking module is further used to integrate the angular velocity to obtain the posture orientation of the camera, and integrate the acceleration to obtain the speed of the camera in response to the second selection condition not being met; calculate the second posture of the camera based on the posture orientation of the camera and the speed of the camera; and perform posture estimation based on the second posture of the camera to obtain a tracking result.

[0153] In one embodiment of the device, the device also includes: a calibration module, which is used to obtain zero-speed attitude data obtained by the IMU at zero speed in response to satisfying a zero-speed calibration condition, and the zero-speed calibration condition includes: a first difference between the IMU acceleration and the gravity acceleration in the IMU is less than a preset difference threshold, a first variance between the IMU acceleration and the gravity acceleration is less than a preset variance threshold, and a variance of the de-zeroed angular velocity is less than the variance threshold; a state error is calculated based on the zero-speed attitude data and the tracking result; the angular velocity zero bias in the IMU is recalculated, and the roll angle and pitch angle are calibrated using the projection of the gravity acceleration; and the attitude data recorded by the IMU is calibrated using Kalman filtering and the state error.

[0154] In one embodiment of the device, the multiple frame images are panoramic images, and the method further includes: a fusion module, used to obtain multiple target maps obtained by multiple terminal devices; determine the initial position in each target map, and relocate according to the panoramic image and the initial position to determine the position of each target map; based on the position of each target map, perform map fusion on the multiple target maps.

[0155] Each module in the above-mentioned map construction device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0156] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store multiple frames of images and map data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a map construction method is implemented.

[0157] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a map construction method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0158] Those skilled in the art will understand that Fig.10 The structure shown in or 11 is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the computer device to which the scheme of the present disclosure is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0159] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0160] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0161] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0162] It should be noted that the multi-frame images and target maps involved in the present disclosure are information and data authorized by the user or fully authorized by all parties.

[0163] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present disclosure may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided by the present disclosure may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0164] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above-described embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the attached claims.

Claims

1. A map construction method, characterized in that: The method comprises: Obtain multiple frames of images captured during the mapping process; Calculating the motion amplitude of the camera shooting the multiple frames of images, and calculating the image brightness change of the multiple frames of images; According to the characteristic characteristics of adjacent images in the multiple frames of images, the motion amplitude and the image brightness change, a matching tracking method is selected for tracking to obtain a tracking result, wherein the tracking method includes: a feature tracking method, a direct tracking method and an IMU tracking method; A map is constructed according to the tracking results to obtain map elements, and a loop is performed according to the feature points of the multiple frames of images to obtain a target map.

2. The method according to claim 1, characterized in that The characteristic characteristics include: characteristic quantity and characteristic quality; the matching tracking method is selected for tracking according to the characteristic characteristics of adjacent images in the multi-frame images, the motion amplitude and the image brightness change to obtain the tracking result, including: In response to satisfying a first selection condition of the feature tracking method, selecting the feature tracking method for tracking to obtain a tracking result, wherein the first selection condition includes: the number of features is greater than a preset feature number threshold, and the quality of the features is greater than a preset quality index; In response to satisfying the second selection condition of the direct tracking method, selecting the direct tracking method for tracking to obtain a tracking result; the second selection condition includes: the first selection condition is not satisfied, and the motion amplitude is less than a preset amplitude threshold, and the image luminance change is less than a preset change threshold; In response to the second selection condition not being met, an IMU tracking method is selected for tracking to obtain a tracking result.

3. The method according to claim 2, characterized in that The calculating of the motion amplitude of the camera used to shoot the multiple frames of images includes: For each frame of the multiple frames of images, determining a feature point in each frame of the image; Perform feature matching on the feature points in each frame of the image to obtain the matching results; At least calculating the first pose of the camera in each frame of the image based on the matching results and using epipolar geometry; Determine the camera's motion amplitude based on the first pose of the camera in each frame of the image; or, Obtaining the posture data recorded by the IMU in the camera; The posture data is integrated to obtain a second posture of the camera, and the movement amplitude of the camera is determined according to the second posture.

4. The method according to claim 3, characterized in that: In response to satisfying the first selection condition of the feature tracking method, selecting the feature tracking method for tracking to obtain a tracking result includes: In response to satisfying the first selection condition of the feature tracking method, acquiring the posture data recorded by the IMU in the camera; Time-aligning the timestamp of the posture data recorded by the IMU with the timestamp of the multiple frames of images; Based on the time-aligned posture data and the first posture of the camera, posture estimation is performed to obtain a tracking result, wherein the first posture is calculated based on multiple frames of images.

5. The method according to claim 3, characterized in that: The posture data includes: angular velocity and acceleration; in response to the second selection condition not being met, selecting the IMU tracking method for tracking, and obtaining a tracking result, including: In response to the second selection condition not being met, integrating the angular velocity to obtain the attitude orientation of the camera, and integrating the acceleration to obtain the velocity of the camera; Calculating a second posture of the camera based on the posture orientation of the camera and the speed of the camera; Perform posture estimation according to the second posture of the camera to obtain a tracking result.

6. The method according to any one of claims 3 to 5, characterized in that The method further comprises: In response to satisfying a zero-speed calibration condition, obtaining zero-speed attitude data obtained by the IMU at zero speed, wherein the zero-speed calibration condition includes: a first difference between the IMU acceleration and the gravity acceleration in the IMU is less than a preset difference threshold, a first variance between the IMU acceleration and the gravity acceleration is less than a preset variance threshold, and a variance of the de-zeroed angular velocity is less than the variance threshold; Calculating a state error according to the zero-speed posture data and the tracking result; Recalculate the angular velocity zero bias in the IMU, and calibrate the roll angle and pitch angle using the projection of the gravitational acceleration; The attitude data recorded by the IMU is calibrated using Kalman filtering and the state error.

7. The method according to claim 1, characterized in that The multiple frames of images are panoramic images, and the method further includes: Obtain multiple target maps from multiple terminal devices; Determine an initial position in each target map, and reposition according to the panoramic image and the initial position to determine the position of each target map; Based on the location of each target map, the multiple target maps are fused.

8. A map construction device, characterized in that: The device comprises: An image acquisition module is used to acquire multiple frames of images captured during the mapping process; A calculation module, used to calculate the motion amplitude of the camera used to shoot the multiple frames of images, and to calculate the image brightness change of the multiple frames of images; A tracking module, used for selecting a matching tracking method for tracking according to the characteristic characteristics of adjacent images in the multiple frames of images, the motion amplitude and the image brightness change, to obtain a tracking result, wherein the tracking method includes: a feature tracking method, a direct tracking method and an IMU tracking method; The map construction module is used to construct a map according to the tracking results to obtain map elements, and to perform loop closure according to the feature points of the multiple frames of images to obtain a target map.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images