Point cloud completion method, system, device, and storage medium
By reconstructing and stitching feature maps of images acquired by LiDAR, and using neural networks for point cloud completion, the problems of slow computing speed and high memory requirements in existing technologies are solved, achieving a more efficient point cloud completion effect and improving the safety of vehicle-road cooperative systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TIANYI TRANSPORTATION TECH CO LTD
- Filing Date
- 2023-07-06
- Publication Date
- 2026-06-02
AI Technical Summary
Existing point cloud completion technologies suffer from slow computation speed, reliance on the geometric symmetry of the completion target, high memory requirements for computing devices, and poor completion results, which affect the recall and accuracy of point cloud data collected by LiDAR in vehicle-road cooperative systems.
The reconstructed point cloud is obtained by reconstructing the acquired image. Feature maps of the target region and the region to be completed are extracted and stitched together to complete the point cloud. The feature map is calculated and decoded using a neural network to avoid dependence on geometric symmetry and only complete the calculation for a part of the region.
It improves the accuracy and robustness of point cloud completion, reduces unnecessary computation, lowers the memory space requirements of computing devices, and increases the model calculation speed.
Smart Images
Figure CN116823666B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of vehicle-road cooperation, specifically to a point cloud completion method, system, device, and storage medium. Background Technology
[0002] In intelligent vehicle-road cooperative systems, point cloud data collected by LiDAR is often used for 3D object detection. However, due to limitations in the internal structure of the LiDAR itself (such as the small vertical field of view) and external factors (such as occlusion), some objects in the point cloud data collected by LiDAR are missing. This seriously affects the recall and accuracy of 3D object detection models and poses a significant safety hazard to traffic participants in vehicle-road cooperative systems. Therefore, point cloud completion is of great practical significance for completing the point cloud data collected by LiDAR in vehicle-road cooperative systems.
[0003] Existing point cloud completion techniques typically perform indiscriminate completion on point cloud data acquired by a single-modality LiDAR. These point cloud completion techniques usually suffer from problems such as slow computation speed, reliance on the geometric symmetry of the completion target, high memory requirements of computing devices, and poor completion results. Summary of the Invention
[0004] In view of this, in order to overcome at least one aspect of the above problems, embodiments of the present invention propose a point cloud completion method, comprising the following steps:
[0005] The acquired image is reconstructed to obtain a reconstructed point cloud, and the target region is obtained based on the reconstructed point cloud;
[0006] Acquire the point cloud to be completed and determine the region to be completed in the point cloud;
[0007] The target region is subjected to feature extraction to obtain a first feature map, and the region to be completed is subjected to feature extraction to obtain a second feature map;
[0008] The third feature map is obtained by concatenating the first feature map with the second feature map;
[0009] Point cloud completion is performed on the region to be completed based on the third feature map.
[0010] In some embodiments, reconstructing the acquired image to obtain a reconstructed point cloud further includes:
[0011] The acquired images are input into the first feature extraction network for semantic feature extraction;
[0012] The extracted semantic features are compressed using a first neural network;
[0013] Perform a preset number of deconvolutions on the compressed semantic features;
[0014] The reconstructed point cloud is obtained by reconstructing the result of the last deconvolution.
[0015] In some embodiments, and based on the reconstructed point cloud, the target region is obtained, further comprising:
[0016] The reconstructed point cloud is mapped from a top-down perspective to obtain a first bird's-eye view;
[0017] The target region is obtained by segmenting the first bird's-eye view using a semantic segmentation model.
[0018] In some embodiments, acquiring the point cloud to be completed and determining the region to be completed in the point cloud further includes:
[0019] A second bird's-eye view is obtained by performing top-down view mapping on the point cloud to be completed;
[0020] Calculate the intersection of the target area in the first bird's-eye view and the second bird's-eye view;
[0021] The region to be completed is determined by reverse mapping the regions corresponding to the intersection in the second bird's-eye view.
[0022] In some embodiments, feature extraction of the target region to obtain a first feature map further includes:
[0023] The target region is input into a second feature extraction network for semantic feature extraction;
[0024] The extracted semantic features are compressed using a second neural network;
[0025] The compressed semantic features are reconstructed into the first feature map.
[0026] In some embodiments, feature extraction of the region to be completed to obtain a second feature map further includes:
[0027] The point cloud of the region to be completed is input into a third neural network for semantic feature extraction to obtain a second feature map.
[0028] In some embodiments, point cloud completion of the region to be completed based on the third feature map further includes:
[0029] The fourth feature map is obtained by using the fourth neural network to calculate the third feature map;
[0030] The fifth feature map is obtained by performing point-level max pooling on the fourth feature map;
[0031] The fifth feature map is decoded using a fifth neural network to obtain the completed point cloud corresponding to the region to be completed.
[0032] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a point cloud completion system, comprising:
[0033] The first processing module is configured to reconstruct the acquired image to obtain a reconstructed point cloud, and obtain the target region based on the reconstructed point cloud;
[0034] The second processing module is configured to acquire the point cloud to be completed and determine the region to be completed in the point cloud;
[0035] The third processing module is configured to extract features from the target region to obtain a first feature map and extract features from the region to be completed to obtain a second feature map;
[0036] The splicing calculation module is configured to splice the first feature map and the second feature map to obtain a third feature map;
[0037] The fourth processing module is configured to perform point cloud completion on the region to be completed based on the third feature map.
[0038] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a computer device, comprising:
[0039] At least one processor; and
[0040] The memory stores a computer program that can run on the processor, and when the processor executes the program, it performs the steps of any of the point cloud completion methods described above.
[0041] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of any of the point cloud completion methods described above.
[0042] The present invention has one of the following beneficial technical effects: The proposed solution completes point cloud data by collecting image data features, avoiding dependence on the geometric symmetry of the completion target, improving the accuracy of point cloud completion, and achieving better point cloud completion results. Simultaneously, it only performs completion calculations on a portion of the original point cloud to be completed, significantly reducing invalid computation, lowering the memory space requirements of computing devices, and improving the model calculation speed. Attached Figure Description
[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.
[0044] Figure 1 A flowchart illustrating the point cloud completion method provided in an embodiment of the present invention;
[0045] Figure 2 A schematic diagram of the structure of the Encode-Image-A module provided in an embodiment of the present invention;
[0046] Figure 3 A flowchart illustrating a point cloud completion method provided in an embodiment of the present invention;
[0047] Figure 4 A schematic diagram of the structure of the Encode-Image-B module provided in an embodiment of the present invention;
[0048] Figure 5 A schematic diagram of the point cloud completion system provided in an embodiment of the present invention;
[0049] Figure 6 A schematic diagram of the structure of a computer device provided for an embodiment of the present invention;
[0050] Figure 7 A schematic diagram of the structure of a computer-readable storage medium provided for an embodiment of the present invention. Detailed Implementation
[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.
[0052] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.
[0053] According to one aspect of the present invention, embodiments of the present invention provide a point cloud completion method, such as... Figure 1 As shown, it may include the following steps:
[0054] S1, reconstruct the acquired image to obtain a reconstructed point cloud, and obtain the target region based on the reconstructed point cloud;
[0055] S2, acquire the point cloud to be completed and determine the region to be completed in the point cloud;
[0056] S3, perform feature extraction calculation on the target region to obtain a first feature map and perform feature extraction calculation on the region to be completed to obtain a second feature map;
[0057] S4, the first feature map and the second feature map are processed to obtain the third feature map;
[0058] S5, perform point cloud completion on the region to be completed based on the third feature map.
[0059] The proposed solution adds global semantic information of the target to be completed to the acquired image data, making point cloud completion no longer dependent on the geometric symmetry of the target, thus improving the accuracy and robustness of point cloud completion and achieving better point cloud completion results. Furthermore, by performing completion calculations only on a portion of the point cloud, it significantly reduces unnecessary computation, lowers the memory requirements of computing devices, and improves model computation speed.
[0060] In some embodiments, S1, reconstructing the acquired image to obtain a reconstructed point cloud, further includes:
[0061] The acquired images are input into the first feature extraction network for semantic feature extraction;
[0062] The extracted semantic features are compressed using a first neural network;
[0063] Perform a preset number of deconvolutions on the compressed semantic features;
[0064] The reconstructed point cloud is obtained by reconstructing the result of the last deconvolution.
[0065] In some embodiments, the Encode-Image-A module can be used to process the input image, such as... Figure 2 As shown, the image can first be input into a first feature extraction network (such as ResNet34, Darknet53, etc.) for high-dimensional semantic feature extraction. Then, the output high-dimensional semantic features are compressed using a first neural network (such as a first MLP, Multi-layer Perception) to obtain a one-dimensional feature map. For example, the one-dimensional feature map can be 1*12800. Then, K deconvolutions (2D Deconvolution, DeConv2D) are performed on the feature map. The value of K is in the range of [1, 10], which can be selected according to the actual situation. Finally, the reshape function is used to calculate the W*3 point cloud from the last deconvolution result, where W is the number of points in the reconstructed point cloud, and 3 indicates that the coordinate dimension of each point is 3, i.e. (x, y, z).
[0066] In some embodiments, S1, and obtaining the target region based on the reconstructed point cloud, further includes:
[0067] The reconstructed point cloud is mapped from a top-down perspective to obtain a first bird's-eye view;
[0068] The target region is obtained by segmenting the first bird's-eye view using a semantic segmentation model.
[0069] Specifically, such as Figure 3 As shown, images captured from a camera can be input into the Encode-Image-A module to reconstruct a 3D point cloud. Then, the reconstructed point cloud is mapped from a top-down perspective to obtain a bird's-eye view (BEV). A semantic segmentation model is used to segment the BEV image to obtain the Region of Interest (ROI), which is the target region. The semantic segmentation model used can be DANet, DeepLabv3, Segment Anything, OCNet, etc. Since vehicles in a vehicle-to-everything (V2X) system only care about the presence of obstacles, vehicles, pedestrians, etc., in the drivable area during operation, in this embodiment of the invention, the target region is selected as the drivable area of the vehicle.
[0070] In some embodiments, S2, acquiring the point cloud to be completed and determining the region to be completed in the point cloud to be completed, further includes:
[0071] A second bird's-eye view is obtained by performing top-down view mapping on the point cloud to be completed;
[0072] Determine the intersection between the target area of the first bird's-eye view and the second bird's-eye view;
[0073] The region to be completed is determined by reverse mapping the regions corresponding to the intersection in the second bird's-eye view.
[0074] Specifically, the area to be completed can be determined based on the target area, such as... Figure 3 As shown, the original point cloud to be completed, acquired by LiDAR, can be mapped from a top-down perspective to obtain a second bird's-eye view of the original point cloud to be completed. The intersection (Intersect) of this second bird's-eye view with the target region is then calculated to obtain the Region of Interest (ROI) of the original point cloud under the BEV (Browser Image Value) perspective. Finally, the ROI region under the BEV perspective of the original point cloud is reverse-mapped to calculate the region to be completed, i.e., the point cloud ROI region. In this way, by utilizing both the image BEV and the point cloud BEV, the point cloud ROI region is obtained without prior knowledge or a point cloud map for ROI region extraction, thus avoiding reliance on high-precision point cloud maps.
[0075] In some embodiments, S3, extracting features from the target region to obtain a first feature map, further includes:
[0076] The target region is input into a second feature extraction network for semantic feature extraction;
[0077] The extracted semantic features are compressed using a second neural network;
[0078] The compressed semantic features are reconstructed into the first feature map.
[0079] Specifically, such as Figure 3 As shown, the target region of the 3D point cloud BEV view reconstructed from the image is input into the Encode-Image-B module to calculate the first feature map.
[0080] In some embodiments, feature extraction can be performed using the Encode-Image-B module, such as... Figure 4 As shown, the target region of the reconstructed point cloud obtained from the image under the BEV view can be input into the second feature extraction network (such as ResNet34, Darknet53, etc.) for high-dimensional feature extraction. Then, the output high-dimensional semantic features are compressed using a second neural network (such as a second MLP) to reduce the dimensionality of the high-dimensional semantic features. Finally, the compressed semantic results are used to calculate the first feature map of N*H using the reshape function, where N is the number of points in the feature map and H is the feature vector length of each point in the feature map.
[0081] In some embodiments, S2, performing feature extraction on the region to be completed to obtain a second feature map, further includes:
[0082] The point cloud of the region to be completed is input into a third neural network for semantic feature extraction to obtain a second feature map.
[0083] Specifically, such as Figure 3 As shown, the point cloud within the region to be completed in the original point cloud can be input into a third neural network (e.g., a third MLP, where the number of layers in an MLP is generally an integer greater than 1 and less than 5) to extract semantic features and obtain an N*M second feature map, where N is the number of points in the feature map and M is the feature vector length of each point in the feature map.
[0084] In some embodiments, point cloud completion of the region to be completed based on the third feature map further includes:
[0085] The fourth feature map is obtained by using the fourth neural network to calculate the third feature map;
[0086] The fifth feature map is obtained by performing point-level max pooling on the fourth feature map;
[0087] The fifth feature map is decoded using a fifth neural network to obtain the completed point cloud corresponding to the region to be completed.
[0088] Specifically, such as Figure 3 As shown, the first and second feature maps are concatenated using the concatenation function `cat()` to obtain a third feature map of N*(H+M). A fourth neural network (e.g., a fourth MLP) is then used to calculate this third feature map, also of N*(H+M). This calculated fourth feature map contains higher-dimensional and more abstract semantic features than the third feature map, resulting in richer global information and improved accuracy in point cloud completion. Point-wise maxpooling is then performed on the fourth feature map to obtain a fifth feature map of 1*(H+M), which is obtained by maximizing the 0th dimension of the N*(H+M) fourth feature map. This fifth feature map contains all the semantic feature information of the objects in the point cloud region to be completed. Finally, a fifth neural network (e.g., a fifth MLP) is used to decode the fifth feature map to obtain the predicted completed point cloud set. Since vehicles in a vehicle-road cooperative system only care about whether there are obstacles, vehicles, pedestrians, or other objects in the drivable area during their journey, the proposed solution only performs point cloud completion on the area to be completed, rather than completing all point cloud data collected by LiDAR. Using this calculation strategy can significantly reduce unnecessary computation of the model and improve the computational efficiency of point cloud completion.
[0089] The proposed solution completes point cloud data using acquired image data, avoiding reliance on the geometric symmetry of the completion target, thus improving the accuracy of point cloud completion and achieving better completion results. Furthermore, by performing completion calculations only on a portion of the point cloud, it significantly reduces unnecessary computation, lowers the memory requirements of computing devices, and improves model computation speed.
[0090] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a point cloud completion system 400, such as... Figure 5 As shown, it includes:
[0091] The first processing module 401 is configured to reconstruct the acquired image to obtain a reconstructed point cloud, and to obtain a target region based on the reconstructed point cloud;
[0092] The second processing module 402 is configured to acquire the point cloud to be completed and determine the region to be completed in the point cloud to be completed.
[0093] The third processing module 403 is configured to extract features from the target region to obtain a first feature map and extract features from the region to be completed to obtain a second feature map.
[0094] The splicing calculation module 404 is configured to splice the first feature map and the second feature map to obtain a third feature map;
[0095] The fourth processing module 405 is configured to perform point cloud completion on the region to be completed based on the third feature map.
[0096] In some embodiments, the first processing module 401 is further configured to:
[0097] The acquired images are input into the first feature extraction network for semantic feature extraction;
[0098] The extracted semantic features are compressed using a first neural network;
[0099] Perform a preset number of deconvolutions on the compressed semantic features;
[0100] The reconstructed point cloud is obtained by reconstructing the result of the last deconvolution.
[0101] In some embodiments, the first processing module 401 is further configured to:
[0102] The reconstructed point cloud is mapped from a top-down perspective to obtain a first bird's-eye view;
[0103] The target region is obtained by segmenting the first bird's-eye view using a semantic segmentation model.
[0104] In some embodiments, the second processing module 402 is further configured to:
[0105] A second bird's-eye view is obtained by performing top-down view mapping on the point cloud to be completed;
[0106] Determine the intersection between the target area of the first bird's-eye view and the second bird's-eye view;
[0107] The region to be completed is determined by reverse mapping the regions corresponding to the intersection in the second bird's-eye view.
[0108] In some embodiments, the third processing module 403 is further configured to:
[0109] The target region is input into a second feature extraction network for semantic feature extraction;
[0110] The extracted semantic features are compressed using a second neural network;
[0111] The first feature map is reconstructed from the compressed semantic features.
[0112] In some embodiments, the third processing module 403 is further configured to:
[0113] The point cloud of the region to be completed is input into a third neural network for semantic feature extraction to obtain a second feature map.
[0114] In some embodiments, the fourth processing module 405 is further configured to:
[0115] The fourth feature map is obtained by using the fourth neural network to calculate the third feature map;
[0116] The fifth feature map is obtained by performing point-level max pooling on the fourth feature map;
[0117] The fifth feature map is decoded using a fifth neural network to obtain the completed point cloud corresponding to the region to be completed.
[0118] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 6 As shown, embodiments of the present invention also provide a computer device 501, comprising:
[0119] At least one processor 520; and
[0120] The memory 510 stores a computer program 511 that can run on the processor. When the processor 520 executes the program, it performs the steps of any of the point cloud completion methods described above.
[0121] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 7 As shown, embodiments of the present invention also provide a computer-readable storage medium 601, which stores a computer program 610. When the computer program 610 is executed by a processor, it performs the steps of any of the point cloud completion methods described above.
[0122] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods.
[0123] Furthermore, it should be understood that the computer-readable storage medium (e.g., memory) described herein may be volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory.
[0124] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, blocks, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.
[0125] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.
[0126] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.
[0127] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0128] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0129] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.
Claims
1. A point cloud completion method, characterized in that, Includes the following steps: The acquired image is reconstructed to obtain a reconstructed point cloud, and the target region is obtained based on the reconstructed point cloud; Acquire the point cloud to be completed and determine the region to be completed in the point cloud; The target region is subjected to feature extraction to obtain a first feature map, and the region to be completed is subjected to feature extraction to obtain a second feature map; The first feature map and the second feature map are concatenated to obtain the third feature map; Point cloud completion is performed on the region to be completed based on the third feature map; The acquired image is reconstructed to obtain a reconstructed point cloud, and the target region is obtained based on the reconstructed point cloud, further including: The acquired images are input into the first feature extraction network for semantic feature extraction; The extracted semantic features are compressed using a first neural network; Perform a preset number of deconvolutions on the compressed semantic features; The reconstructed point cloud is obtained by reconstructing the result of the last deconvolution. The reconstructed point cloud is mapped from a top-down perspective to obtain a first bird's-eye view; The target region is obtained by semantically segmenting the first bird's-eye view using a semantic segmentation model. Acquiring the point cloud to be completed and determining the region to be completed in the point cloud, further includes: A second bird's-eye view is obtained by performing top-down view mapping on the point cloud to be completed; Calculate the intersection of the target area in the first bird's-eye view and the second bird's-eye view; The region to be completed is determined by reverse mapping the regions corresponding to the intersection in the second bird's-eye view.
2. The method as described in claim 1, characterized in that, The first feature map is obtained by extracting features from the target region, and the process further includes: The target region is input into a second feature extraction network for semantic feature extraction; The extracted semantic features are compressed using a second neural network; The compressed semantic features are reconstructed into the first feature map.
3. The method as described in claim 1, characterized in that, The second feature map obtained by feature extraction of the region to be filled further includes: The point cloud of the region to be completed is input into a third neural network for semantic feature extraction to obtain a second feature map.
4. The method as described in claim 1, characterized in that, Point cloud completion of the region to be completed based on the third feature map further includes: The fourth feature map is obtained by using the fourth neural network to calculate the third feature map; The fifth feature map is obtained by performing point-level max pooling on the fourth feature map; The fifth feature map is decoded using a fifth neural network to obtain the completed point cloud corresponding to the region to be completed.
5. A point cloud completion system, characterized in that, include: The first processing module is configured to reconstruct the acquired image to obtain a reconstructed point cloud, and obtain the target region based on the reconstructed point cloud; The second processing module is configured to acquire the point cloud to be completed and determine the region to be completed in the point cloud; The third processing module is configured to extract features from the target region to obtain a first feature map and extract features from the region to be completed to obtain a second feature map; The splicing calculation module is configured to splice the first feature map and the second feature map to obtain a third feature map; The fourth processing module is configured to perform point cloud completion on the region to be completed based on the third feature map; The first processing module is also configured as follows: The acquired images are input into the first feature extraction network for semantic feature extraction; The extracted semantic features are compressed using a first neural network; Perform a preset number of deconvolutions on the compressed semantic features; The reconstructed point cloud is obtained by reconstructing the result of the last deconvolution. The reconstructed point cloud is mapped from a top-down perspective to obtain a first bird's-eye view; The target region is obtained by semantically segmenting the first bird's-eye view using a semantic segmentation model. The second processing module is also configured as follows: A second bird's-eye view is obtained by performing top-down view mapping on the point cloud to be completed; Calculate the intersection of the target area in the first bird's-eye view and the second bird's-eye view; The region to be completed is determined by reverse mapping the regions corresponding to the intersection in the second bird's-eye view.
6. A computer device, comprising: At least one processor; as well as A memory storing a computer program executable on the processor, characterized in that the processor executes the program by performing the steps of the method as described in any one of claims 1-4.
7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it performs the steps of the method as described in any one of claims 1-4.