Obstacle detection method and apparatus, electronic device, and storage medium

Abstract

The application discloses an obstacle detection method and device, electronic equipment and a storage medium. The application is used for accurately identifying the category and motion state of an obstacle, and then realizing strategic obstacle avoidance. In the embodiment of the application, first, a target point cloud corresponding to an obstacle of continuous multiple frames of images containing a target scene is acquired; then, for each obstacle of a latter frame of image of any two adjacent frames of image, based on the target point cloud corresponding to the obstacle of the latter frame of image and the target point cloud corresponding to each obstacle of a former frame of image, the distance between the obstacle of the latter frame of image and each obstacle of the former frame of image is determined; finally, the shortest distance in the determined distance is compared with a preset distance, and according to the comparison result, the state of the obstacle is determined, wherein the state includes dynamic and static.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent robots, and particularly relates to an obstacle detection method and device, electronic equipment and a storage medium. BACKGROUND

[0002] With the development of science and technology, in recent years, intelligent mobile robots are increasingly popular in life, and typical examples include self-driving cars, logistics robots, service robots, and household sweeping and cleaning robots. Autonomous obstacle avoidance capability in complex environments is an important embodiment of the intelligence of mobile robots, and obstacle detection of the surrounding environment is the basis for realizing autonomous obstacle avoidance. Obstacle detection refers to obtaining environmental data based on a camera, a laser radar, and the like, processing the data through an algorithm, and then obtaining information of potential obstacles. However, in the related art, the category and motion state of the obstacle cannot be accurately identified, and thus strategic obstacle avoidance cannot be achieved. SUMMARY

[0003] The present application aims to provide an obstacle detection method, device, electronic equipment and storage medium for accurately identifying the category and motion state of the obstacle.

[0004] In a first aspect, an obstacle detection method is provided, comprising:

[0005] obtaining a plurality of continuous frames of images containing a target scene, performing cluster analysis on point cloud clusters of obstacles in the plurality of continuous frames of images to obtain target point clouds corresponding to the obstacles; wherein the point cloud clusters of the obstacles are obtained by performing semantic segmentation on the plurality of continuous frames of images;

[0006] For each obstacle in a later frame of image of any two adjacent frames of image, determining a distance between the obstacle in the later frame of image and each obstacle in an earlier frame of image based on the target point cloud corresponding to the obstacle in the later frame of image and the target point cloud corresponding to each obstacle in the earlier frame of image;

[0007] comparing the shortest distance in the determined distances with a preset distance, and determining a state of the obstacle according to a comparison result, wherein the state includes dynamic and static.

[0008] In the present application, since the dynamic obstacle is in a moving state in the target scene, the position of the dynamic obstacle is different in each frame, resulting in different distances between the same dynamic obstacle in the two adjacent frames of image. Therefore, by comparing the distances of the obstacles in the two adjacent frames, the motion state of the obstacle is determined, so that the intelligent mobile robot can determine whether the current obstacle is in a static state or a moving state. Furthermore, the intelligent mobile robot can be controlled to perform strategic obstacle avoidance according to the motion state of the obstacle, thereby improving the accuracy of obstacle recognition of the intelligent mobile robot.

[0009] In one possible embodiment, the acquiring the continuous multiple frames of images containing a target scene, performing clustering analysis on point cloud clusters of obstacles in the continuous multiple frames of images to obtain target point clouds corresponding to the obstacles, comprises:

[0010] Based on the continuous multiple frames of images, a segmentation result is obtained by a semantic segmentation method, the segmentation result comprising point cloud clusters corresponding to the ground and point cloud clusters corresponding to each obstacle; the point cloud clusters comprising a first point cloud corresponding to each obstacle and a category of each obstacle;

[0011] Based on point clouds in an intermediate point cloud having the same coordinates as the first point cloud, the point cloud set is obtained, wherein the intermediate point cloud is obtained by a preprocessing method based on the continuous multiple frames of images;

[0012] Performing clustering analysis on the point cloud set to obtain multiple clustering results, wherein the clustering results comprise a second point cloud corresponding to each obstacle;

[0013] Comparing the second point cloud with each of the first point clouds and determining target point clouds corresponding to each obstacle according to the comparison results.

[0014] In the present application, the first point cloud is obtained by a semantic segmentation method, which fully utilizes the advantage that semantic segmentation can meet real-time requirements, and at the same time, a clustering analysis method with more accurate results is used, so that the point cloud obtained in the present application can meet real-time requirements and has high accuracy.

[0015] In one possible embodiment, the comparing the second point cloud with each of the first point clouds and determining target point clouds corresponding to each obstacle according to the comparison results comprises:

[0016] For any one second point cloud, comparing the any one second point cloud with each of the first point clouds corresponding to each obstacle in turn to determine a first point cloud having the highest similarity with the second point cloud;

[0017] Marking points in the second point cloud having the same coordinates as the first point cloud having the highest similarity as having the same category as the first point cloud having the highest similarity, and marking points in the second point cloud having different coordinates from the first point cloud having the highest similarity as a general category.

[0018] In the present application, the accuracy of the point cloud corresponding to each obstacle is further improved according to the comparison results of the first point cloud and the second point cloud.

[0019] In one possible embodiment, after the comparing the second point cloud with each of the first point clouds and determining target point clouds corresponding to each obstacle according to the comparison results, the method further comprises:

[0020] Screening the point cloud corresponding to the obstacle, removing the point cloud corresponding to the obstacle whose point number is less than a preset value.

[0021] By screening the point cloud according to the number of points in the point cloud, the interference items with a small number of points in the point cloud can be filtered out, and the difficulty of subsequent calculation is reduced.

[0022] In one possible embodiment, before obtaining the point cloud set based on the point cloud with the same coordinates as the intermediate point cloud and the first point cloud, the method further comprises:

[0023] Obtaining a plurality of continuous frames of images, wherein each frame of image comprises a color image and a depth image;

[0024] Preprocessing the depth image included in the plurality of continuous frames of images to obtain an intermediate point cloud;

[0025] The intermediate point cloud is obtained based on the plurality of continuous frames of images by a semantic segmentation method, comprising:

[0026] Inputting the color image and the depth image included in the plurality of continuous frames of images into a semantic segmentation model to obtain a segmentation result.

[0027] Different images of a target scene are processed differently, which can improve accuracy compared to a method of processing only one kind of image, and makes the present application more universal.

[0028] In one possible embodiment, the preprocessing the depth image included in the plurality of continuous frames of images to obtain an intermediate point cloud comprises:

[0029] Converting the depth image into point cloud information in a target coordinate system;

[0030] Voxel down-sampling the point cloud information to obtain sampled point cloud information;

[0031] Determining points in the sampled point cloud information with extreme values in each dimension coordinate as target points; the extreme values include maximum values and minimum values;

[0032] Performing denoising processing on a region determined by the target points to obtain an intermediate point cloud corresponding to an obstacle of the plurality of continuous frames of images.

[0033] In the embodiments of the present application, the amount of calculation is reduced by the method of voxel down-sampling the point cloud information, and the waste of resources is avoided; and by the denoising method, the influence of noise on obstacle recognition is avoided.

[0034] In one possible embodiment, after determining the state of the obstacle, the method further comprises:

[0035] performing motion estimation on the dynamic obstacle to obtain a motion speed of the dynamic obstacle.

[0036] In the embodiments of the present application, the motion speed of the obstacle can be accurately determined through motion estimation on the dynamic obstacle, so that the intelligent mobile robot is more accurate when performing strategic obstacle avoidance.

[0037] In one possible embodiment, for each obstacle in a later frame of any two adjacent frames of images, after determining the distance between the obstacle in the later frame and each obstacle in the previous frame based on the point cloud corresponding to the obstacle in the later frame and the point cloud corresponding to each obstacle in the previous frame, the method further comprises:

[0038] constructing a similarity distance matrix according to the distance between the obstacle in the later frame and each obstacle in the previous frame; each column in the similarity distance matrix is the distance between the obstacle in the later frame and each obstacle in the previous frame;

[0039] calculating the minimum value of each column in the similarity distance matrix;

[0040] taking the minimum value as the shortest distance.

[0041] In the embodiments of the present application, the shortest distance between the obstacles in the two frames of images can be quickly obtained by constructing a similarity distance matrix, thereby reducing the amount of calculation.

[0042] In some possible embodiments, the determining the state of the obstacle according to the comparison result comprises:

[0043] if the comparison result shows that the shortest distance is less than or equal to the preset distance, the state of the obstacle is static;

[0044] if the comparison result shows that the shortest distance is greater than the preset distance, the state of the obstacle is dynamic.

[0045] The second aspect of the present application further provides an obstacle detection device, which comprises:

[0046] an acquisition module configured to acquire a plurality of continuous frames of images containing a target scene, perform clustering analysis on point cloud clusters of obstacles in the plurality of continuous frames of images, and obtain target point clouds corresponding to the obstacles; wherein the point cloud clusters of the obstacles are obtained through semantic segmentation on the plurality of continuous frames of images;

[0047] a distance determination module configured to determine, for each obstacle in a later image of any two adjacent images, a distance between the obstacle in the later image and each obstacle in an earlier image based on a target point cloud corresponding to the obstacle in the later image and a target point cloud corresponding to each obstacle in the earlier image;

[0048] a comparison module configured to compare the shortest distance in the determined distances with a preset distance, and determine a state of the obstacle according to a comparison result, wherein the state includes dynamic and static.

[0049] In an embodiment, the obtaining module is configured to obtain continuous multiple images containing a target scene, and perform clustering analysis on point cloud clusters of obstacles in the continuous multiple images to obtain target point clouds corresponding to the obstacles.

[0050] obtain a segmentation result by a semantic segmentation method based on the continuous multiple images, wherein the segmentation result includes a point cloud cluster corresponding to a ground and a point cloud cluster corresponding to each obstacle; and the point cloud cluster includes a first point cloud corresponding to each obstacle and a category of each obstacle.

[0051] obtain the point cloud set based on point clouds in an intermediate point cloud having the same coordinates as the first point cloud, wherein the intermediate point cloud is obtained by a preprocessing method based on the continuous multiple images.

[0052] perform clustering analysis on the point cloud set to obtain multiple clustering results, wherein the clustering results include a second point cloud corresponding to each obstacle.

[0053] compare the second point cloud with each first point cloud, and determine a target point cloud corresponding to each obstacle according to a comparison result.

[0054] In an embodiment, the obtaining module is configured to compare the second point cloud with each first point cloud, and determine a target point cloud corresponding to each obstacle according to a comparison result.

[0055] for any one second point cloud, compare the any one second point cloud with each first point cloud corresponding to each obstacle in turn, and determine a first point cloud having the highest similarity with the second point cloud.

[0056] mark points in the second point cloud having the same coordinates as the first point cloud having the highest similarity as the same category as the first point cloud having the highest similarity, and mark points in the second point cloud having different coordinates as a general category.

[0057] In an embodiment, after the acquisition module performs comparison between the second point cloud and each of the first point clouds, and determines the target point cloud corresponding to each of the obstacles according to the comparison result, the cargo module is further configured to:

[0058] filtering the target point cloud corresponding to the obstacle, and removing the target point cloud corresponding to the obstacle whose number of points is less than a preset value.

[0059] In an embodiment, before the acquisition module performs obtaining the point cloud set based on the point cloud in the intermediate point cloud which has the same coordinates as the first point cloud, the acquisition module is further configured to:

[0060] acquiring a plurality of continuous frames of images, wherein each frame of image includes a color image and a depth image;

[0061] preprocessing the depth image included in the plurality of continuous frames of images to obtain an intermediate point cloud;

[0062] the semantic segmentation method based on the plurality of continuous frames of images to obtain a segmentation result, including:

[0063] inputting the color image and the depth image included in the plurality of continuous frames of images into a semantic segmentation model to obtain a segmentation result.

[0064] In an embodiment, when the acquisition module performs preprocessing the depth image included in the plurality of continuous frames of images to obtain an intermediate point cloud, it is configured to:

[0065] convert the depth image into point cloud information in a target coordinate system;

[0066] perform voxel down-sampling on the point cloud information to obtain sampled point cloud information;

[0067] determine points in the sampled point cloud information whose coordinates in each dimension are extreme values as target points; the extreme values include maximum values and minimum values;

[0068] perform denoising processing on the region determined by the target points to obtain an intermediate point cloud corresponding to the obstacle of the plurality of continuous frames of images.

[0069] In an embodiment, after the comparison module determines the state of the obstacle, the device further comprises:

[0070] a speed determination module configured to perform motion estimation on the dynamic obstacle to obtain a motion speed of the dynamic obstacle.

[0071] In one embodiment, the distance determining module determines, for each obstacle in the latter image of any two adjacent images, a distance between the obstacle in the latter image and each obstacle in the former image based on a point cloud corresponding to the obstacle in the latter image and a point cloud corresponding to each obstacle in the former image, and the apparatus further comprises:

[0072] a shortest distance determining module configured to construct a similarity distance matrix according to the distances between the obstacle in the latter image and each obstacle in the former image; each column in the similarity distance matrix is a distance between the obstacle in the latter image and each obstacle in the former image;

[0073] calculating a minimum value of each column in the similarity distance matrix;

[0074] taking the minimum value as the shortest distance.

[0075] In one embodiment, the comparing module determines, according to the comparison result, a state of the obstacle, including:

[0076] if the comparison result shows that the shortest distance is less than or equal to the preset distance, the state of the obstacle is static;

[0077] if the comparison result shows that the shortest distance is greater than the preset distance, the state of the obstacle is dynamic.

[0078] In a third aspect, another embodiment of the present application further provides an electronic device, including at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method provided in the first aspect of the present application.

[0079] In a fourth aspect, another embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is used to make a computer execute any method provided in the first aspect of the present application.

[0080] Other features and advantages of the present application will be further described in the following description, and will become apparent from the description, or will be learned through practice of the present application. The objects and other advantages of the present application will be realized and achieved by particularly pointed out in the written specification, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0081] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced as follows. Obviously, the drawings to be introduced below only represent some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0082] Figure 1 The application scenario diagram of the obstacle detection method provided by the embodiments of the present application is shown in the following figure.

[0083] Figure 2 The overall flowchart of the obstacle detection method provided by the embodiments of the present application is shown in the following figure.

[0084] Figure 3 The flowchart of obtaining the target point cloud of the obstacle detection method provided by the embodiments of the present application is shown in the following figure.

[0085] Figure 4 The schematic diagram of voxel down-sampling of the obstacle detection method provided by the embodiments of the present application is shown in the following figure.

[0086] Figure 5 The similar distance matrix schematic diagram of the obstacle detection method provided by the embodiments of the present application is shown in the following figure.

[0087] Figure 6 The overall flowchart of the obstacle detection method provided by the embodiments of the present application is shown in the following figure.

[0088] Figure 7 The device schematic diagram of the obstacle detection method provided by the embodiments of the present application is shown in the following figure.

[0089] Figure 8 The electronic device schematic diagram of the obstacle detection method provided by the embodiments of the present application is shown in the following figure. DETAILED DESCRIPTION

[0090] In order to make the ordinary person skilled in the art better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings.

[0091] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and do not necessarily represent a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0092] The inventors' research revealed that with technological advancements, intelligent mobile robots have become increasingly prevalent in daily life in recent years, including self-driving cars, logistics robots, service robots, and household cleaning robots. Autonomous obstacle avoidance in complex environments is a crucial aspect of mobile robot intelligence, and obstacle detection in the surrounding environment is fundamental to achieving this. Obstacle detection involves acquiring environmental data using cameras, LiDAR, etc., processing this data through algorithms to obtain information about potential obstacles. However, current technologies often fail to accurately identify the type and movement state of obstacles, thus hindering strategic obstacle avoidance.

[0093] In view of this, this application proposes an obstacle detection method, apparatus, electronic device, and storage medium to solve the above-mentioned problems. The inventive concept of this application can be summarized as follows: First, acquire the target point cloud corresponding to the obstacles in a series of consecutive frames of images containing the target scene; then, for each obstacle in the later frame of any two adjacent frames, determine the distance between the obstacle in the later frame and each obstacle in the previous frame based on the target point cloud corresponding to the obstacle in the later frame and the target point cloud corresponding to each obstacle in the previous frame; finally, compare the shortest distance among the determined distances with a preset distance, and determine the state of the obstacle based on the comparison result, wherein the state includes dynamic and static.

[0094] like Figure 1 The diagram shown illustrates an application scenario of the obstacle detection method according to an embodiment of this application. The diagram includes: a terminal device 10, a server 20, and a memory 30; wherein:

[0095] Server 20 can simultaneously process multiple consecutive frames of images containing the target scene acquired by terminal devices 101, 102, 103, etc., to obtain the target point cloud corresponding to the obstacles in the multiple consecutive frames. Then, for each obstacle in the later frame of any two adjacent frames, based on the target point cloud corresponding to the obstacle in the later frame and the target point cloud corresponding to each obstacle in the previous frame, the distance between the obstacle in the later frame and each obstacle in the previous frame is determined. Finally, the shortest distance among the determined distances is compared with a preset distance, and the state of the obstacle is determined according to the comparison result, where the state includes dynamic and static.

[0096] After identifying obstacles in the target scene, the state of the obstacles can be saved in the memory for later use.

[0097] The description in this application is only detailed for a single server or terminal device, but those skilled in the art should understand that the terminal device 10, the server 20 and the memory 30 shown are intended to represent the operation of the terminal device, the server and the memory involved in the technical solution of the present application. The single server and the memory are detailed at least for the convenience of illustration, and not to imply a limitation on the number, type or location of the terminal device and the server. It should be noted that if additional modules are added to the illustrated environment or individual modules are removed therefrom, the underlying concept of the example embodiments of the present application will not change. In addition, although the application environment shown in the figures is a single server and a single terminal device, those skilled in the art should understand that the application environment can also include multiple servers and multiple terminal devices. Figure 1 The bidirectional arrow from the memory 30 to the server 20 is shown in the figure, but those skilled in the art can understand that the transmission and reception of the above-mentioned data also needs to be realized through the network.

[0098] It should be noted that the memory in the embodiments of the present application can be a cache system, a hard disk storage, a memory storage, etc. In addition, the obstacle detection method proposed in the present application is not only applicable to the application scenarios shown, but also applicable to any device with obstacle detection needs. Figure 1

[0099] In order to facilitate the understanding of the obstacle detection method proposed in the present application, the obstacle detection method provided by the embodiments of the present application will be described in detail below in conjunction with the drawings.

[0100] As shown in Figure 2 , the obstacle detection method provided by the embodiments of the present application includes the following steps:

[0101] Step 201: Obtain continuous multiple frames of images containing a target scene, and perform clustering analysis on point cloud clusters of obstacles in the continuous multiple frames of images to obtain target point clouds corresponding to the obstacles; wherein the point cloud clusters of the obstacles are obtained by performing semantic segmentation on the continuous multiple frames of images;

[0102] Step 202: For each obstacle of a later frame of image in any two adjacent frames of image, determine the distance between the obstacle of the later frame of image and each obstacle of the former frame of image based on the target point cloud corresponding to the obstacle of the later frame of image and the target point cloud corresponding to each obstacle of the former frame of image.

[0103] Step 203: Compare the shortest distance in the determined distances with a preset distance, and determine the state of the obstacle according to the comparison result, wherein the state includes dynamic and static. Static means that the obstacle is in a static state in the target scene; dynamic means that the obstacle is in a motion state in the target scene.

[0104] In some embodiments, obtaining the target point clouds of the obstacles corresponding to the continuous multiple frames of images containing the target scene can be implemented as the step shown in Figure 3 .​

[0105] In step 301, based on the continuous multiple frames of images, a segmentation result is obtained by a semantic segmentation manner, the segmentation result including a point cloud cluster corresponding to the ground and a point cloud cluster corresponding to each obstacle; the point cloud cluster including a first point cloud corresponding to each obstacle and a category of each obstacle.

[0106] In an embodiment, first, continuous multiple frames of images are acquired, wherein each frame of image includes a color image and a depth image; the color image and the depth image are input into a semantic segmentation model to obtain a segmentation result. In the embodiment of the present application, the semantic segmentation model can be any one of the following deep learning networks: Separation-and-Aggregation Gate RGB-D Convolutional Neural Network (SA-GateNet), Malleable 2.5D Convolutional Neural Network (2.5D CNN), etc. In the present application, in order to make the running speed of the semantic segmentation model faster, an efficient convolutional neural network (Efficient Convolutional Neural Networks for Mobile Vision Applications, MobileNet) is preferably applied to a light network as a benchmark network for deep convolution feature extraction to meet the real-time requirement.

[0107] Since the semantic segmentation model in the present application is trained in advance with an obstacle set labeled with obstacle categories as samples, the segmentation result output after the continuous multiple frames of images containing a target scene are input into the semantic segmentation model contains a point cloud cluster corresponding to the ground and a point cloud cluster corresponding to each obstacle; the point cloud cluster including a first point cloud corresponding to each obstacle and a category of each obstacle. The category of the obstacle can include: a table, a chair, a garbage can, etc. All objects identified in the target scene are obstacles.

[0108] In step 302, based on the point cloud in the intermediate point cloud with the same coordinates as the first point cloud, a point cloud set is obtained, wherein the intermediate point cloud is obtained by a preprocessing manner based on the continuous multiple frames of images.

[0109] In one embodiment, a plurality of continuous frames of images are acquired, wherein each frame of image comprises a color image and a depth image; the depth images included in the plurality of continuous frames of images are preprocessed to obtain intermediate point clouds. The preprocessing can be implemented as follows: first, the depth image is converted into point cloud information in a target coordinate system; in order to avoid the point cloud processed being too dense, resulting in large amount of computation and slow computation speed, the point cloud information is voxel down-sampled to obtain sampled point cloud information; wherein the voxel down-sampling is to create a three-dimensional voxel grid according to the input point cloud, and for each voxel, the center of gravity of all points in the voxel is used to represent the voxel; as shown in FIG. 3, the resolution of the voxel down-sampling is set as L, i.e. the point cloud is divided into n L*L*L cubes, and for each cube, the center of gravity of the cube is used to represent all points in the cube, thereby obtaining the intermediate point cloud. Figure 4

[0110] The points in the sampled point cloud information that have extreme values in each dimension coordinate are determined as target points; the extreme values include maximum and minimum values; in the embodiment of the present application, the coordinates of the points in the point cloud information are three-dimensional coordinates, and the target points are the points with maximum and minimum coordinates in the x-axis, the points with maximum and minimum coordinates in the y-axis, and the points with maximum and minimum coordinates in the z-axis. In one embodiment, there can be two points A with coordinate X and B with coordinate X that have the maximum coordinate in the x-axis, and in this case, the coordinates of the points A and B in the y-axis and z-axis are compared, and if the coordinate Y1 of the point A in the y-axis is greater than the coordinate Y2 of the point B in the y-axis, and the coordinate Z1 of the point A in the z-axis is less than the coordinate Z2 of the point B, then the coordinate of the point with the maximum coordinate in the x-axis is (X, Y1, Z2).

[0111] The region determined by the target points is denoised to obtain the point cloud corresponding to the obstacles of the plurality of continuous frames of images. In the embodiment of the present application, in order to reduce the noise of the edges of the obstacles, neighborhood radius filtering is performed on the region, and the specific implementation is as follows: taking the current point as the center and lr as the radius, if the number of point clouds in the radius range is less than a preset number, the current point is filtered out, otherwise it is retained. It should be noted that other methods of removing noise points are also applicable to the present application, and will not be described here.

[0112] In one embodiment, in order to further improve the efficiency of identifying obstacles under the premise of meeting real-time, in the embodiment of the present application, the point cloud set is clustered in step 303 to obtain a plurality of clustering results, wherein the clustering results include a second point cloud corresponding to each obstacle;

[0113] In step 304, the second point cloud and each first point cloud are compared, and the target point cloud corresponding to each obstacle is determined according to the comparison result.

[0114] ​In some embodiments, for any one of the second point clouds, each of the second point clouds is compared with each of the first point clouds corresponding to the obstacles in turn to determine the first point cloud with the highest similarity to the second point cloud; points in the second point cloud that have the same coordinates as the first point cloud with the highest similarity are marked as the same category as the first point cloud with the highest similarity, and points in the second point cloud that have different coordinates from the first point cloud with the highest similarity are marked as a general category. For example, the first point cloud corresponding to each obstacle is C1 (category 1), C2 (category 2), C3 (category 3), and so on; the second point cloud corresponding to each obstacle is P1, P2, P3, and so on; for any one of P1, P2, P3, and so on, taking P1 as an example, P1 is compared with C1, C2, C3, and so on in turn to determine that the first point cloud with the highest similarity to P1 is C2, then P1 is compared with C2, points in P1 that have the same coordinates as C2 are marked as category 2 and are denoted as O1, and points in P1 that have different coordinates from C2 are marked as the general category; then for P2, P2 is compared with C1, C2, C3, and so on in turn to determine that the first point cloud with the highest similarity to P2 is C1, then P2 is compared with C1, points in P2 that have the same coordinates as C1 are marked as category 1 and are denoted as O2, and points in P2 that have different coordinates from C1 are also marked as the general category. Finally, the target point cloud O1, O2, O3, and so on corresponding to each obstacle can be obtained.

[0115] In some embodiments, in order to reduce the waste of computing resources and make the calculation result more accurate, in the embodiments of the present application, after the target point cloud O1, O2, O3, and so on corresponding to each obstacle is obtained, the target point cloud corresponding to the obstacle can be screened to remove the target point cloud corresponding to the obstacle whose number of points is less than a preset value. For example, as shown in FIG. 8, O1 has 20 points, O2 has 28 points, and O3 has 4 points, and the preset value is 10, so O3 is removed. Through this method, the influence of the point cloud with low reference value on the detection result is avoided. Figure 3

[0116] In the embodiments of the present application, for each obstacle in the latter one of any two adjacent frames of images, after the distance between the obstacle in the latter one of the two frames of images and each obstacle in the former one of the two frames of images is determined based on the target point cloud corresponding to the obstacle in the latter one of the two frames of images and the target point cloud corresponding to each obstacle in the former one of the two frames of images, in order to quickly determine the state of the obstacle, in the embodiments of the present application, a similar distance matrix can be constructed to determine the shortest distance between the obstacle in the latter one of the two frames of images and each obstacle in the former one of the two frames of images. Specifically, the method can be implemented as follows:

[0117] ​A similarity distance matrix is constructed according to the distances between the obstacles in the latter image and each obstacle in the former image; each column in the similarity distance matrix is the distance between the obstacle in the latter image and each obstacle in the former image; the minimum value of each column in the similarity distance matrix is calculated; and the minimum value is the shortest distance. For example: for obstacles A, B and C in the latter image, assuming that there are obstacles E, F and G in the former image; the distances between A and E, F and G are calculated, then the distances between B and E, F and G are calculated; and then the distances between C and E, F and G are calculated; the calculation results constitute a matrix D as shown in Figure 5 For each column in the matrix D, the minimum value of each column is calculated, that is, the shortest distance. The shortest distance is compared with a preset distance, if the shortest distance is less than or equal to the preset distance, it is a static obstacle, otherwise it is a dynamic obstacle. Figure 5

[0118] After determining the state of the obstacle, in order to more accurately identify the speed of the obstacle, motion estimation is performed on the dynamic obstacle to determine the speed of the obstacle. In the embodiment of the application, Kalman filtering is used to perform motion estimation on the obstacle to obtain the speed of the obstacle. It should be noted that other methods for estimating motion are also applicable to the application and will not be described here.

[0119] For ease of understanding, the overall flow of the obstacle detection method proposed in the application is described in detail as follows: Figure 6

[0120] In step 601, color images and depth images of a target scene are obtained.

[0121] In step 602, the depth images are converted into point cloud information in a target coordinate system.

[0122] In step 603, the point cloud information is down-sampled by voxel to obtain sampled point cloud information.

[0123] In step 604, points in the sampled point cloud information with extreme values in each dimension coordinate are determined as target points.

[0124] In step 605, the region determined by the target points is denoised to obtain point clouds corresponding to obstacles in continuous multiple frames of images, that is, intermediate point clouds.

[0125] In step 606, the color images and depth images included in the continuous multiple frames of images are input into a semantic segmentation model to obtain a segmentation result; the segmentation result includes a point cloud cluster corresponding to the ground and a point cloud cluster corresponding to each obstacle; the point cloud cluster includes a first point cloud corresponding to each obstacle and a category of each obstacle.

[0126] ​​In step 607: based on the point cloud in the intermediate point cloud that is the same as the coordinates of the first point cloud, obtain a point cloud set;

[0127] In step 608: perform clustering analysis on the point cloud set to obtain a plurality of clustering results, wherein the clustering results include a second point cloud corresponding to each obstacle;

[0128] In step 609: for any one second point cloud, sequentially compare the any one second point cloud with each first point cloud corresponding to each obstacle to determine a first point cloud with the highest similarity to the second point cloud;

[0129] In step 610: mark points in the second point cloud that are the same as the coordinates in the first point cloud with the highest similarity as the same category as the first point cloud with the highest similarity, and mark points in the second point cloud that are different from the coordinates in the first point cloud with the highest similarity as a general category; the marked point cloud is a target point cloud;

[0130] In step 611: filter the target point cloud corresponding to the obstacle to remove the target point cloud corresponding to the obstacle whose point number is less than a preset value;

[0131] In step 612: for each obstacle in a latter image of any two adjacent images, based on the point cloud corresponding to the obstacle of the latter image and the point cloud corresponding to each obstacle of a former image, determine the distance between the obstacle of the latter image and each obstacle of the former image;

[0132] In step 613: construct a similarity distance matrix according to the distance between the obstacle of the latter image and each obstacle of the former image; each column in the similarity distance matrix is the distance between the obstacle of the latter image and each obstacle of the former image;

[0133] In step 614: calculate the minimum value of each column in the similarity distance matrix, and take the minimum value as the shortest distance;

[0134] In step 615: compare the shortest distance in the determined distance with a preset distance, and determine the state of the obstacle according to the comparison result;

[0135] In step 616: perform motion estimation on the dynamic obstacle to obtain the motion speed of the dynamic obstacle.

[0136] As shown in Figure 7 based on the same inventive concept, an obstacle detection device 700 is proposed, comprising:

[0137] The acquisition module 7001 is configured to acquire continuous multiple frames of images containing a target scene, and perform clustering analysis on point cloud clusters of an obstacle in the continuous multiple frames of images to obtain a target point cloud corresponding to the obstacle.

[0138] The distance determination module 7002 is configured to, for each obstacle in a later frame of image of any two adjacent frames of image, determine a distance between the obstacle in the later frame of image and each obstacle in an earlier frame of image based on the target point cloud corresponding to the obstacle in the later frame of image and the target point cloud corresponding to each obstacle in the earlier frame of image.

[0139] The comparison module 7003 is configured to compare the shortest distance in the determined distances with a preset distance, and determine a state of the obstacle according to a comparison result, wherein the state includes dynamic and static.

[0140] In an embodiment, when the acquisition module 7001 performs the acquisition of the target point cloud corresponding to the obstacle of the continuous multiple frames of images containing the target scene, the acquisition module 7001 is configured to:

[0141] Based on the continuous multiple frames of images, a segmentation result is obtained by a semantic segmentation manner, and the segmentation result includes a point cloud cluster corresponding to a ground and a point cloud cluster corresponding to each obstacle; the point cloud cluster includes a first point cloud corresponding to each obstacle and a category of each obstacle.

[0142] Based on a point cloud in an intermediate point cloud that is the same as the coordinates of the first point cloud, the point cloud set is obtained, wherein the intermediate point cloud is obtained by a preprocessing manner based on the continuous multiple frames of images.

[0143] The clustering analysis is performed on the point cloud set to obtain multiple clustering results, wherein the clustering result includes a second point cloud corresponding to each obstacle.

[0144] The second point cloud and each first point cloud are compared, and a target point cloud corresponding to each obstacle is determined according to a comparison result.

[0145] In an embodiment, when the acquisition module 7001 performs the comparison of the second point cloud and each first point cloud and determines the target point cloud corresponding to each obstacle according to a comparison result, the acquisition module 7001 is configured to:

[0146] For any one second point cloud, the any one second point cloud is compared with each first point cloud corresponding to each obstacle in sequence to determine a first point cloud with the highest similarity to the second point cloud.

[0147] Marking the points in the second point cloud that are the same as the coordinates in the first point cloud with the highest similarity as the same category as the first point cloud with the highest similarity, and marking the points that are different from the coordinates in the first point cloud with the highest similarity as a general category.

[0148] In one embodiment, after the acquisition module 7001 performs comparison between the second point cloud and each first point cloud, and determines the target point cloud corresponding to each obstacle according to the comparison result, the device is further configured to:

[0149] Filtering the target point cloud corresponding to the obstacle, and removing the target point cloud corresponding to the obstacle whose number of points in the target point cloud corresponding to the obstacle is less than a preset value.

[0150] In one embodiment, before the acquisition module 7001 performs obtaining the point cloud set based on the point cloud in the intermediate point cloud that is the same as the coordinates of the first point cloud, the acquisition module 7001 is further configured to:

[0151] Acquiring a plurality of continuous frames of images, wherein each frame of image includes a color image and a depth image;

[0152] Preprocessing the depth image included in the plurality of continuous frames of images to obtain an intermediate point cloud;

[0153] The segmentation result obtained based on the plurality of continuous frames of images by the semantic segmentation method includes:

[0154] Inputting the color image and the depth image included in the plurality of continuous frames of images into a semantic segmentation model to obtain a segmentation result.

[0155] In one embodiment, when the acquisition module 7001 performs preprocessing the depth image included in the plurality of continuous frames of images to obtain an intermediate point cloud, it is configured to:

[0156] Convert the depth image into point cloud information in a target coordinate system;

[0157] Perform voxel down-sampling on the point cloud information to obtain sampled point cloud information;

[0158] Determine points in the sampled point cloud information that have extreme values in each dimension coordinate as target points; the extreme values include maximum values and minimum values;

[0159] Performing denoising processing on the region determined by the target points to obtain an intermediate point cloud corresponding to the obstacles of the plurality of continuous frames of images.

[0160] In one embodiment, after the comparison module 7003 performs determination of the state of the obstacle, the device further includes:

[0161] The speed determination module is configured to perform motion estimation on the dynamic obstacle to obtain a motion speed of the dynamic obstacle.

[0162] In one embodiment, after the distance determination module 7002 performs, for each obstacle in a latter image of any two adjacent images, distance determination between the obstacle in the latter image and each obstacle in a former image based on a point cloud corresponding to the obstacle in the latter image and a point cloud corresponding to each obstacle in the former image, the apparatus further comprises:

[0163] The shortest distance determination module 7002 is configured to construct a similarity distance matrix according to the distances between the obstacle in the latter image and each obstacle in the former image; each column in the similarity distance matrix is the distance between the obstacle in the latter image and each obstacle in the former image.

[0164] The minimum value of each column in the similarity distance matrix is calculated.

[0165] The minimum value is taken as the shortest distance.

[0166] In one embodiment, when the comparison module 7003 performs the determination of the state of the obstacle according to the comparison result, it is configured to:

[0167] If the comparison result shows that the shortest distance is less than or equal to the preset distance, the state of the obstacle is static.

[0168] If the comparison result shows that the shortest distance is greater than the preset distance, the state of the obstacle is dynamic.

[0169] After the obstacle detection method and apparatus of the exemplary embodiments of the present application are introduced, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0170] Those skilled in the art can understand that each aspect of the present application can be implemented as a system, a method or a program product. Therefore, each aspect of the present application can be specifically implemented as follows: a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" here.

[0171] In some possible embodiments, the electronic device according to the present application can at least include at least one processor and at least one memory. The memory stores program code, which, when executed by the processor, causes the processor to perform the steps in the obstacle detection method according to various exemplary embodiments of the present application described above in the specification.

[0172] The electronic device 130 according to this embodiment of the present application will be described below with reference to Figure 8 Figure 8 The electronic device 130 shown is merely an example and should not limit the function and usage range of the embodiments of the present application.

[0173] As Figure 8 shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 can include, but are not limited to, at least one processor 131, at least one memory 132, a bus 133 connecting different system components, including the memory 132 and the processor 131. Among them, the processor 131 is configured to execute the obstacle detection method proposed by the embodiments of the present application, and the memory 132 is used to store the detected obstacles.

[0174] The bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor or a local bus using any of a variety of bus structures.

[0175] The memory 132 can include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322, and can further include a read-only memory (ROM) 1323.

[0176] The memory 132 can also include a program / utility 1325 having a set of (at least one) program modules 1324, such as an operating system, one or more application programs, other program modules, and program data, each of which or a combination of which can include an implementation of a network environment.

[0177] ​The electronic device 130 can also communicate with one or more external devices 134 such as a keyboard or a pointing device, through an input / output (I / O) interface 135. And, the electronic device 130 can communicate with one or more devices that enable user interaction with the electronic device 130, and / or one or more devices that enable communication of the electronic device 130 with one or more other electronic devices. This communication can be via the I / O interface 135. The electronic device 130 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the public network, such as the Internet, through a network adapter 136. As depicted, the network adapter 136 is in communication with the other modules of the electronic device 130 through the bus 133. It should be appreciated that other hardware and / or software modules can be used in conjunction with the electronic device 130, including but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc. It should be understood that the various electronic devices 130 described above can not be connected to a network when configured to detect obstacles. The various electronic devices can be added or removed as appropriate for a particular implementation.

[0178] In some possible embodiments, the various aspects of the obstacle detection method provided by the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to perform the steps of the obstacle detection method according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device.

[0179] The program product can employ any combination of one or more computer-readable media. The computer-readable media can be a computer-readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0180] The program product for obstacle detection of the embodiments of the present application can employ a portable compact disc read only memory (CD-ROM) and include a program code, and can be executed on an electronic device. However, the program product of the present application is not limited thereto, and in the present document, a readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0181] The readable signal medium can include a data signal that is propagated in baseband or that is propagated as a carrier wave, in which the readable program code is contained. Such a propagated signal can take a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. The readable signal medium can also be any readable medium that is not a storage medium or that is not medium that is not a storage medium that can be used to store program code momentarily for use by or in connection with an instruction execution system, apparatus, or device.

[0182] The program code contained on the readable medium can be transmitted using any suitable medium, including, but not limited to, wireless, wired, optical fiber cable, RF, etc., or any suitable combination thereof.

[0183] The program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's electronic device, partly on the user's electronic device, as a stand-alone software package, partly on the user's electronic device and partly on a remote electronic device or entirely on the remote electronic device or server. In the latter scenario, the remote electronic device can be connected to the user's electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external electronic device (for example, through the Internet using an Internet Service Provider).

[0184] It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units embodied by multiple units.

[0185] Moreover, although the operations of the method(s) herein can be described in a particular, sequential order, this order is not meant to be a limitation and

[0186] Those of skill in the art would understand that embodiments of the present application can be provided as a method, system, or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0187] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.

[0188] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.

[0189] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.

[0190] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. An obstacle detection method characterized by, The method comprises: Based on the continuous multiple frames of images containing the target scene, the ground and the first point cloud corresponding to each obstacle are obtained by a pre-constructed semantic segmentation model, the category of each obstacle is obtained, the point cloud set is obtained based on the point cloud with the same coordinates as the first point cloud in the intermediate point cloud, the clustering analysis is performed on the point cloud set, the second point cloud corresponding to each obstacle is obtained, the second point cloud and each first point cloud are compared, and the target point cloud corresponding to each obstacle is determined according to the comparison result, wherein the semantic segmentation model is a lightweight network, which is obtained by training based on an obstacle set with marked obstacle categories as samples, and the intermediate point cloud is obtained by pre-processing based on the continuous multiple frames of images. According to the clustering analysis of the point cloud cluster of the obstacle in the continuous multiple frames of images, the target point cloud corresponding to the obstacle is obtained; wherein the point cloud cluster of the obstacle is obtained by performing semantic segmentation on the continuous multiple frames of images. For each obstacle in the latter image of any two adjacent images, the distance between the obstacle in the latter image and each obstacle in the former image is determined based on the target point cloud corresponding to the obstacle in the latter image and the target point cloud corresponding to each obstacle in the former image. The shortest distance in the determined distance is compared with a preset distance, and the state of the obstacle is determined according to the comparison result, wherein the state includes dynamic and static.

2. The method of claim 1, wherein, The comparison of the second point cloud and each first point cloud according to the comparison result to determine the target point cloud corresponding to each obstacle comprises: For any one second point cloud, the any one second point cloud is compared with each first point cloud corresponding to the obstacle in turn to determine the first point cloud with the highest similarity to the second point cloud; The points in the second point cloud with the same coordinates as the first point cloud with the highest similarity are marked as the same category as the first point cloud with the highest similarity, and the points with different coordinates from the first point cloud with the highest similarity are marked as a general category.

3. The method of claim 1, wherein, After the comparison of the second point cloud and each first point cloud according to the comparison result to determine the target point cloud corresponding to each obstacle, the method further comprises: Filtering the target point cloud corresponding to the obstacle, and removing the target point cloud corresponding to the obstacle with the number of points less than a preset value.

4. The method of claim 1, wherein, Before the point cloud set is obtained based on the point cloud with the same coordinates as the first point cloud in the intermediate point cloud, the method further comprises: Obtaining continuous multiple frames of images, wherein each frame of image comprises a color image and a depth image; Preprocessing the depth image included in the continuous multiple frames of images to obtain an intermediate point cloud.

5. The method of claim 4, wherein, The preprocessing of the depth image included in the continuous multiple frames of images to obtain an intermediate point cloud comprises: Converting the depth image into point cloud information in a target coordinate system; Performing voxel downsampling on the point cloud information to obtain sampled point cloud information; Determining the points with extreme values in each dimension coordinate in the sampled point cloud information as target points; the extreme values include maximum and minimum values; The region determined for the target point is denoised to obtain an intermediate point cloud corresponding to the obstacles of the continuous multiple frames of images.

6. The method of claim 1, wherein, After determining the state of the obstacle, the method further comprises: motion estimation is performed on the dynamic obstacle to obtain the motion speed of the dynamic obstacle.

7. The method of claim 1, wherein, For each obstacle of a later frame of image in any two adjacent frames of image, after determining the distance between the obstacle of the later frame of image and each obstacle of a former frame of image based on the target point cloud corresponding to the obstacle of the later frame of image and the target point cloud corresponding to each obstacle of the former frame of image, the method further comprises: a similarity distance matrix is constructed according to the distance between the obstacle of the later frame of image and each obstacle of the former frame of image; each column in the similarity distance matrix is the distance between the obstacle of the later frame of image and each obstacle of the former frame of image; the minimum value of each column in the similarity distance matrix is calculated; the minimum value is taken as the shortest distance.

8. The method of claim 1, wherein, The state of the obstacle is determined according to the comparison result, comprising: if the comparison result shows that the shortest distance is less than or equal to the preset distance, the state of the obstacle is static; if the comparison result shows that the shortest distance is greater than the preset distance, the state of the obstacle is dynamic.

9. An obstacle detection device characterized by comprising: The device comprises: an acquisition module, configured to obtain a first point cloud corresponding to each obstacle and a category of each obstacle based on a continuous multiple frames of images containing a target scene through a pre-constructed semantic segmentation model, obtain a point cloud set based on a point cloud with the same coordinates as an intermediate point cloud and the first point cloud, perform clustering analysis on the point cloud set to obtain a second point cloud corresponding to each obstacle, compare the second point cloud and each first point cloud, and determine a target point cloud corresponding to each obstacle according to a comparison result, wherein the semantic segmentation model is a lightweight network and is obtained by training based on an obstacle set with marked obstacle categories as samples, and the intermediate point cloud is obtained through a preprocessing manner based on the continuous multiple frames of images; a distance determination module, configured to determine the distance between each obstacle of a later frame of image and each obstacle of a former frame of image in any two adjacent frames of image based on a target point cloud corresponding to the obstacle of the later frame of image and a target point cloud corresponding to each obstacle of the former frame of image; a comparison module, configured to compare the shortest distance in the determined distance with a preset distance and determine the state of the obstacle according to a comparison result, wherein the state comprises dynamic and static.

10. An electronic device, comprising: The computer storage medium stores a computer program, and the computer program is used to make a computer execute the method in any one of claims 1-8.

11. A computer storage medium, characterized in that The computer storage medium stores a computer program, and the computer program is used to make a computer execute the method in any one of claims 1-8.

Images