Target detection method, device, terminal device and computer-readable storage medium

Abstract

The present application provides a target detection method, apparatus, terminal device and computer-readable storage medium. The method comprises: obtaining point cloud data to be detected, the point cloud data to be detected is point cloud data derived from a scene to be detected, and the scene to be detected includes a target to be detected; inputting the point cloud data to be detected into a trained first neural network model so that the first neural network model identifies each point cloud point of the target surface of the target to be detected in the point cloud data to be detected; and determining the size of the target surface based on each point cloud point belonging to the target surface. The present application can solve the technical problem that the size of the target object cannot be accurately detected in the prior art to a certain extent.

Description

Technical Field

[0001] The present application belongs to the technical field of target detection, and in particular, relates to a target detection method, a target detection device, a terminal device, and a computer-readable storage medium. Background Art

[0002] Currently, many fields need to detect the features of target objects in the scene, such as detecting the size of the target object or whether the target object is a scrollable shape.

[0003] At present, the method for detecting the size of the target object is usually a point cloud plane fitting algorithm, such as the RANSAC algorithm. When determining the size of the target object, the point cloud plane fitting algorithm needs to pre-set a position range, determine the point cloud points within the position range as points on the same surface, and then implement surface size detection based on the three-dimensional coordinates of each point cloud point on the same surface. Obviously, for target objects with large surface convexities, the point cloud plane fitting algorithm cannot accurately detect the point cloud points on the same surface, and therefore, cannot accurately detect the size of the surface.

[0004] In addition, in some cases, it is necessary to identify the shape of the target object. Currently, most shape recognition relies entirely on manual work. For example, if the target object is a parcel, when the parcel enters the transfer yard, in order to prevent parcels with shapes that do not meet the requirements (such as spheres, cylinders, etc.) from entering the cross-belt sorting, it is necessary to manually identify the shape of each parcel before the parcel enters the cross-belt sorting, and then select the parcels with shapes that do not meet the requirements (such as spheres, cylinders, etc.). Therefore, the current recognition of the shape of the target object relies entirely on manual work, which is inefficient. Summary of the invention

[0005] In view of this, the present application provides a target detection method, a target detection device, a terminal device and a computer-readable storage medium, which can solve the technical problems in the prior art that the size of the target object cannot be accurately detected and the efficiency of the target object shape recognition is low.

[0006] The first aspect of the present application provides a target detection method, comprising:

[0007] Acquire point cloud data to be detected, where the point cloud data to be detected is point cloud data derived from a scene to be detected, and the scene to be detected includes a target to be detected;

[0008] Inputting the above-mentioned point cloud data to be detected into the trained first neural network model, so that the first neural network model recognizes each point cloud point belonging to the target surface of the above-mentioned target to be detected in the above-mentioned point cloud data to be detected;

[0009] The size of the target surface is determined according to each point cloud point belonging to the target surface.

[0010] A second aspect of an embodiment of the present application provides a target detection device, including:

[0011] A point cloud acquisition module to be detected is used to acquire point cloud data to be detected, where the point cloud data to be detected is point cloud data derived from a scene to be detected, and the scene to be detected includes a target to be detected;

[0012] A target surface detection module, used for inputting the above-mentioned point cloud data to be detected into the trained first neural network model, so that the first neural network model recognizes each point cloud point belonging to the target surface of the above-mentioned target to be detected in the above-mentioned point cloud data to be detected;

[0013] The surface size determination module is used to determine the size of the target surface according to each point cloud point belonging to the target surface.

[0014] In addition to the target detection method provided in the first aspect above, the third aspect of the present application further provides another target detection method, which includes:

[0015] Acquire point cloud data to be detected, where the point cloud data to be detected is derived from point cloud data of a scene to be detected, where the scene to be detected includes one or more target objects;

[0016] Inputting the above-mentioned point cloud data to be detected into the trained second neural network model, so that the second neural network model recognizes the shape of the target object to which each point cloud point in the above-mentioned point cloud data to be detected belongs;

[0017] If the ratio of the number of point cloud points whose shape is a preset shape belongs to the target object is greater than the preset ratio to the number of all point cloud points in the above-mentioned point cloud data to be detected, it is confirmed that the above-mentioned scene to be detected contains a target object of the above-mentioned preset shape.

[0018] A fourth aspect of the present application provides a target detection device, comprising:

[0019] A point cloud acquisition module to be detected is used to acquire point cloud data to be detected, where the point cloud data to be detected is derived from point cloud data of a scene to be detected, where the scene to be detected includes one or more target objects;

[0020] A shape detection module, used for inputting the above-mentioned point cloud data to be detected into the trained second neural network model, so that the second neural network model recognizes the shape of the target object to which each point cloud point in the above-mentioned point cloud data to be detected belongs;

[0021] The preset shape determination module is used to confirm that the above-mentioned scene to be detected contains a target object of the above-mentioned preset shape if the proportion of the number of point cloud points whose shape is the preset shape to all point cloud points of the above-mentioned point cloud data to be detected is greater than the preset proportion.

[0022] The fifth aspect of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the above memory and executable on the above processor, wherein the above processor implements the steps of the above-mentioned first aspect method or the third aspect method when executing the above-mentioned computer program.

[0023] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect or the method of the third aspect are implemented.

[0024] The seventh aspect of the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the first aspect method or the third aspect method.

[0025] As can be seen from the above, the target detection method provided by the first aspect of the present application is to identify each point cloud point belonging to the target surface through a trained neural network model. When the training samples are sufficient, the trained neural network model can accurately identify each point cloud point belonging to the same surface, avoiding the need for the traditional point cloud plane fitting algorithm to identify points located on the same surface by pre-setting the position range. Therefore, the solution provided by the present application can more accurately identify each point cloud point belonging to the same surface, so that the target surface size of the target to be detected can be more accurately detected to a certain extent. Therefore, the present application can solve the technical problem that the size of the target object cannot be accurately detected in the prior art to a certain extent.

[0026] The target detection method provided in the third aspect of the present application utilizes a neural network model to identify the shape of the target object to which each point cloud point belongs, and then determines whether there is a target object of a preset shape in the scene to be detected. When the training samples are sufficient, the trained neural network model can accurately identify the shape of the target object to which each point cloud point belongs, thereby further accurately detecting whether there is a target object of a preset shape in the scene to be detected. Therefore, the target detection method can automatically identify whether there is a target object of a preset shape in the scene to be detected without having to rely on manual labor, thereby improving the recognition efficiency of the target object shape to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 This is a schematic diagram of the implementation process of the target detection method provided in Example 1 of the present application;

[0029] Figure 2 This is a schematic diagram of an application scenario of the target detection method provided in Example 1 of the present application;

[0030] Figure 3 This is a schematic diagram of the implementation process of another target detection method provided in Example 2 of the present application;

[0031] Figure 4 This is a schematic diagram of an application scenario of another target detection method provided in Example 2 of the present application;

[0032] Figure 5 is a schematic diagram of the structure of the target detection device provided in Example 3 of the present application;

[0033] Figure 6 is a schematic diagram of the structure of another target detection device provided in Embodiment 4 of the present application;

[0034] Figure 7 It is a structural diagram of the terminal device provided in Example 5 of the present application. DETAILED DESCRIPTION

[0035] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0036] The target detection method provided in the embodiment of the present application is applicable to a terminal device. By way of example, the terminal device includes but is not limited to: a desktop computer, a notebook, a tablet computer, a smart phone and other computing devices.

[0037] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0038] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0039] In addition, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0040] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0041] Embodiment 1

[0042] The target detection method provided in the first embodiment of the present application is described below. Figure 1 , the target detection method includes:

[0043] In step S101, point cloud data to be detected is obtained, where the point cloud data to be detected is derived from point cloud data of a scene to be detected, where the scene to be detected includes a target to be detected;

[0044] In the embodiment of the present application, before determining the size of the target to be detected, the point cloud data to be detected must first be obtained. The point cloud data to be detected can be: point cloud data of the scene to be detected containing the target to be detected; or can also be: point cloud data obtained by preprocessing the point cloud data of the scene to be detected containing the target to be detected. It is easy for those skilled in the art to understand that in order to accurately detect the target surface size of the above-mentioned target to be detected, the above-mentioned point cloud data to be detected should contain the complete target surface. Otherwise, if the above-mentioned point cloud data to be detected only contains part of the target surface, it is obviously impossible to reasonably detect the size of the target surface.

[0045] The above-mentioned preprocessing includes downsampling the point cloud data of the scene to be detected (downsampling can be performed using a voxel grid filtering algorithm), denoising (denoising can be performed using a statistical filtering algorithm) and / or removing point cloud points in the point cloud data of the scene to be detected that are not the above-mentioned targets to be detected.

[0046] The following uses the attached Figure 2 , the process of acquiring the point cloud data to be detected is described.

[0047] Attached Figure 2 A schematic diagram of an application scenario of the target detection method provided in Example 1 of the present application is described. Figure 2As shown, the target to be detected is a package 202 located on a conveyor belt surface 201, and there is a terminal device 203 above the package 202, and the terminal device 203 includes a depth image acquisition device 2031 (for example, a depth camera). In order to accurately measure the size of the target surface of the package 202, such as the size of the upper surface, the depth image acquired by the depth image acquisition device 2031 includes the complete upper surface of the package 202 (usually, the depth image acquired by the depth image acquisition device 2031 should include the upper surface and side surface of the package and the belt surface around the package). The terminal device 203 is used to implement steps S101-S103 described in the first embodiment of the present application. The specific process of the terminal device 203 acquiring the point cloud data to be detected is first described below.

[0048] First, the terminal device 203 obtains a depth image including the entire upper surface of the package 202 acquired by the depth image acquisition device 2031;

[0049] Second, the terminal device 203 converts each pixel point in the depth image into each point cloud point to obtain point cloud data of the scene to be detected, wherein the point cloud data of the scene to be detected is composed of each point cloud point obtained by the conversion;

[0050] Third, the terminal device 203 can first adopt the pass-through filtering (PassThrough A filter) algorithm is used to determine the cutoff ranges in the X and Y directions, and the cutoff range in the Z direction is determined according to the installation height of the depth image acquisition device 2031, so as to remove the point cloud points that do not belong to the package 202 in the point cloud data of the scene to be detected, that is, to remove the point cloud points belonging to the belt surface (those skilled in the art will easily understand that, under normal circumstances, it is not possible to completely remove each point cloud point that does not belong to the package 202), and obtain point cloud data 1; then, in order to ensure the execution speed of the subsequent step S102, a voxel grid filtering algorithm can be used to downsample the point cloud data 1, wherein the voxel grid size used in the voxel grid filtering algorithm can be adjusted according to the actual installation height of the depth image acquisition device 2031 (for example, the higher the installation height of the depth image acquisition device 2031, the smaller the voxel grid, and the lower the installation height, the larger the voxel grid, so as to ensure the accuracy of the subsequent surface size calculation), and obtain the downsampled point cloud data 2; then, the noise points of the point cloud data 2 can be removed by a statistical filtering algorithm to ensure the accuracy of the subsequent surface size calculation, and obtain the point cloud data to be detected.

[0051] In step S102, the point cloud data to be detected is input into a trained first neural network model, so that the first neural network model recognizes each point cloud point belonging to the target surface of the target to be detected in the point cloud data to be detected;

[0052] In an embodiment of the present application, the above-mentioned first neural network model may be a pointnet model. The present application does not limit the specific type of the first neural network model. Any neural network model that can process point cloud data may be the above-mentioned first neural network model. For example, the first neural network model may also be a convolutional neural network CNN model that can process three-dimensional data.

[0053] It is easy for a person skilled in the art to understand that before executing step S102, the first neural network model needs to be trained to obtain a trained first neural network model, so that the trained neural network model can identify whether each point cloud point in the input point cloud data belongs to the target surface.

[0054] In the attached Figure 2 In the example shown, the target surface of this step may be the upper surface of the package 202, as discussed below in the attached Figure 2 In the example shown, the training process of the first neural network model for identifying the upper surface of the package can be:

[0055] First, a plurality of sample depth images are collected by using the depth image acquisition device 2031, and then the plurality of sample depth images are processed respectively to obtain sample point cloud data to be detected corresponding to each sample depth image (the specific process of obtaining the sample point cloud data to be detected from the sample depth image can refer to the “specific process of the terminal device 203 obtaining the point cloud data to be detected” described in the above step S101);

[0056] Secondly, for each sample point cloud data to be detected, the label of each point cloud point in the sample point cloud data to be detected is manually marked. The label of each point cloud point can be: upper surface of the package, side surface of the package or belt surface (although the point cloud points that do not belong to the package, that is, the point cloud points corresponding to the belt surface, will be removed when obtaining the sample point cloud data to be detected, but the point cloud points belonging to the belt surface cannot be completely removed. Therefore, the belt surface can be introduced into the model training as a category of labels).

[0057] Then, based on the above-mentioned sample point cloud data to be detected and the label corresponding to each point cloud point in each sample point cloud data to be detected, the first neural network model is trained so that the first neural network model can identify the surface to which each point cloud point in the input point cloud data belongs (that is, whether it belongs to the upper surface of the package, the side surface of the package or the belt surface) with a certain accuracy.

[0058] In step S103, the size of the target surface is determined according to each point cloud point belonging to the target surface;

[0059] Those skilled in the art are aware that each point cloud point in the point cloud data contains three-dimensional coordinates. Therefore, the size of the target surface, such as length, width and / or area, can be determined based on the three-dimensional coordinates of each point cloud point belonging to the target surface.

[0060] In the attached Figure 2 In the example shown, if the image acquisition direction of the depth image acquisition device 2031 is perpendicular to the belt surface 201, the target surface is the upper surface, and the three-dimensional coordinates of each point cloud point in the point cloud data to be detected acquired in step S101 are coordinates in a coordinate system with the image acquisition direction of the depth image acquisition device 2031 as the z-axis, the size of the package 202 can be determined according to the following steps A to C:

[0061] Step A, projecting each point cloud point belonging to the above target surface onto the xoy two-dimensional plane;

[0062] Step B, determining the minimum circumscribed rectangle of each point cloud point projected onto the two-dimensional plane;

[0063] Step C: Determine the length and width of the minimum circumscribed rectangle according to the two-dimensional coordinates of each point cloud point projected onto the xoy two-dimensional plane to obtain the length and width of the target surface.

[0064] That is, firstly, each point cloud point belonging to the upper surface is projected to the xoy plane, and then the convex hull algorithm can be used to determine the minimum circumscribed rectangle of each point cloud point projected to the two-dimensional plane, and finally the length and width of the minimum circumscribed rectangle are determined according to the two-dimensional coordinates of each point cloud point projected to the two-dimensional plane. Considering that all point cloud points belonging to the target surface may not be identified by the first neural network model, and the small number of point cloud points belonging to the target surface in the point cloud data to be detected will lead to deviations in the calculation of the size of the target surface, therefore, the accuracy of measuring the size of the target surface can be further improved by projecting to the two-dimensional plane and determining the minimum circumscribed rectangle.

[0065] In addition, in the attached Figure 2 In the example shown, if the image acquisition direction of the depth image acquisition device 2031 is perpendicular to the belt surface 201, the target surface is the upper surface, and the three-dimensional coordinates of each point cloud point in the point cloud data to be detected acquired in step S101 are coordinates in a coordinate system with the image acquisition direction of the depth image acquisition device 2031 as the z-axis, the height of the package 201 can be determined according to the following steps:

[0066] Step D: obtaining a first distance value between the depth image acquisition device 2031 and the supporting plane (the first distance value is usually pre-stored in the terminal device 203);

[0067] Step E: determining a second distance value between the target plane and the depth image acquisition device according to the point cloud points located on the target surface (i.e., determining the second distance value according to the three-dimensional coordinates of the point cloud points located on the target surface);

[0068] Step F: determine the difference between the first distance value and the second distance value as the height of the package 202.

[0069] In addition, after the above steps C and D, the volume of the package 202 may be determined.

[0070] The target detection method provided in the first embodiment of the present application is to identify each point cloud point belonging to the target surface through a trained neural network model. When the training samples are sufficient, the trained neural network model can accurately identify each point cloud point belonging to the same surface, avoiding the need for the traditional point cloud plane fitting algorithm to identify points located on the same surface by presetting the position range. Therefore, the solution provided by the present application can more accurately identify each point cloud point belonging to the same surface, so that the target surface size of the target to be detected can be more accurately detected to a certain extent. Therefore, the present application can solve the technical problem that the size of the target object cannot be accurately detected in the prior art to a certain extent.

[0071] Embodiment 2

[0072] Another target detection method provided in the second embodiment of the present application is described below. Figure 3 , the target detection method includes:

[0073] In step S301, point cloud data to be detected is obtained, where the point cloud data to be detected is derived from point cloud data of a scene to be detected, and the scene to be detected includes one or more target objects;

[0074] In an embodiment of the present application, the point cloud data to be detected described in step S301 may be: the point cloud data of the above-mentioned scene to be detected; or may be: preprocessing the point cloud data of the above-mentioned scene to be detected, and the point cloud data obtained is the above-mentioned point cloud data to be detected; or may also be: preprocessing the point cloud data of the above-mentioned scene to be detected to obtain preprocessed point cloud data, and then clustering the preprocessed point cloud data to obtain multiple point cloud blocks, and selecting a point cloud block as the above-mentioned point cloud data to be detected.

[0075] Among them, the above-mentioned preprocessing includes downsampling the point cloud data of the above-mentioned scene to be detected (downsampling can be performed using a voxel grid filtering algorithm), denoising (denoising can be performed using a statistical filtering algorithm) and / or removing point cloud points that do not belong to any target object in the point cloud data of the above-mentioned scene to be detected.

[0076] The following uses the attached Figure 4 , the process of acquiring the point cloud data to be detected is described.

[0077] Attached Figure 4 A schematic diagram of an application scenario of the target detection method provided in Example 2 of the present application is described. Figure 4 As shown, the scene to be detected includes three packages located on the belt surface 401, namely a square package 4021, a square package 4022 and a round package 4023. There is a terminal device 403 above the three packages located on the belt surface 401, and the terminal device 403 includes a depth image acquisition device 4031 (for example, a depth camera). The terminal device 403 is used to implement steps S301-S303 described in the second embodiment of the present application. The specific process of the terminal device 403 obtaining the point cloud data to be detected is first described below.

[0078] First, the terminal device 403 obtains the depth image of the scene to be detected acquired by the depth image acquisition device 4031;

[0079] Second, the terminal device 403 converts each pixel point in the depth image into each point cloud point to obtain point cloud data of the scene to be detected, wherein the point cloud data of the scene to be detected is composed of each point cloud point obtained by the conversion;

[0080] Third, the terminal device 403 can first adopt the pass-through filtering (PassThrough filter) algorithm is used to determine the cutoff range in the X and Y directions, and the cutoff range in the Z direction is determined according to the installation height of the depth image acquisition device 4031, so as to remove the point cloud points that do not belong to any package in the point cloud data of the scene to be detected, that is, to remove the point cloud points belonging to the belt surface, and obtain point cloud data 1 (those skilled in the art can easily understand that it may not be possible to completely remove each point cloud point belonging to the belt surface); then, in order to ensure the execution speed of the subsequent step S302, the voxel grid filtering algorithm can be used to downsample the point cloud data 1, wherein the voxel grid size used in the voxel grid filtering algorithm can be adjusted according to the actual installation height of the depth image acquisition device 4031 (for example, the higher the installation height of the depth image acquisition device 4031, the smaller the voxel grid, and the lower the installation height, the larger the voxel grid, so as to ensure the accuracy of the subsequent surface size calculation), and obtain the downsampled point cloud data 2; then, the noise points of the point cloud data 2 can be removed by the statistical filtering algorithm to ensure the accuracy of the shape detection in step S302, and obtain the preprocessed point cloud data;

[0081] Fourth, cluster the preprocessed point cloud data (Euclidean clustering algorithm may be used) to obtain multiple point cloud blocks, and select one point cloud block as the point cloud data to be detected (the purpose of this step is also to improve the execution speed of the subsequent step S302).

[0082] In step S302, the point cloud data to be detected is input into the trained second neural network model, so that the second neural network model recognizes the shape of the target object to which each point cloud point in the point cloud data to be detected belongs;

[0083] In order to enable those skilled in the art to more clearly understand the meaning of “the shape of the target object to which the point cloud points belong”, the following example is used for explanation:

[0084] If the point cloud point X is a point cloud point on the target object Y, then the shape of the target object to which the point cloud point X belongs is the shape of the target object Y;

[0085] If the point cloud point X is not a point cloud point on any target object, the shape of the target object to which the point cloud point X belongs may be a preset shape, for example, a rectangle.

[0086] In an embodiment of the present application, the second neural network model may be a pointnet model. The present application does not limit the specific type of the second neural network model. Any neural network model that can process point cloud data may be the second neural network model. For example, the second neural network model may also be a convolutional neural network (CNN) model that can process three-dimensional data.

[0087] Before step S302, the second neural network model needs to be trained so that the second neural network model can recognize the shape of the target object to which each point cloud point in the input point cloud data belongs.

[0088] In an embodiment of the present application, the label corresponding to each point cloud point may be the shape of the target object to which each point cloud point belongs. For example, the label corresponding to each point cloud point may be: "circular", "rectangular", "cylindrical" (if the second neural network model recognizes that there are point cloud points in the point cloud data to be detected that do not belong to any target object, the second neural network model may uniformly consider the shape of the target object to which the point cloud point belongs to be a certain shape, for example, a rectangle).

[0089] In addition, if the second neural network model identifies that there are point cloud points in the point cloud data to be detected that do not belong to any target object, the second neural network model may not output the label of the point cloud point, or the label of the point cloud point may be "does not belong to any target object". Such solutions are equivalent to the technical solutions of Example 2 of the present application.

[0090] Generally, if the target object is a parcel for transport, parcels with a rolling shape (e.g., spherical or cylindrical) are not allowed to be cross-belt sorted. Therefore, before the parcel is cross-belt sorted, it is necessary to identify whether there are abnormal items in the scene to be detected (parcels with a rolling shape are abnormal items, and the rest are normal items). Figure 4 Taking as an example, the training process of a second neural network model for detecting whether there are abnormal parts in the scene to be detected is discussed:

[0091] First, a plurality of sample depth images are collected by using the depth image acquisition device 4031, and then the plurality of sample depth images are preprocessed and clustered respectively to obtain sample point cloud data to be detected corresponding to each sample depth image (the specific process of obtaining the sample point cloud data to be detected from the sample depth image can refer to the “specific process of the terminal device 403 obtaining the point cloud data to be detected” described in the above step S301);

[0092] Secondly, for each sample point cloud data to be detected, the label of each point cloud point in the sample point cloud data to be detected is manually marked. The label of each point cloud point can be: normal shape, sphere, cylinder (for point cloud points that do not belong to any target object, the label of the point cloud point can be manually marked as normal shape).

[0093] Then, based on the above-mentioned sample point cloud data to be detected and the label corresponding to each point cloud point in each sample point cloud data to be detected, the second neural network model is trained so that the second neural network model can recognize the label corresponding to each point cloud point in the input point cloud data (i.e., normal shape, sphere or cylinder) with a certain accuracy.

[0094] Since normal items usually account for the majority of packages, in order to ensure that the second neural network model does not make misjudgments, more normal items and fewer abnormal items can be used when training the second neural network model.

[0095] In step S303, if the ratio of the number of point cloud points of the target object whose shape is a preset shape to the number of all point cloud points of the point cloud data to be detected is greater than a preset ratio, it is confirmed that the scene to be detected contains a target object of the preset shape;

[0096] Taking the second neural network model capable of detecting whether there are abnormal parts in the scene to be detected as an example, the specific implementation process of step S303 is described:

[0097] If the second neural network model detects that there are 150 point cloud points labeled as normal shape, 150 point cloud points labeled as sphere, and 200 point cloud points labeled as column, assuming that the preset shape is sphere or column, and the preset ratio is 40%, then in step S303, it can be calculated that the number of point cloud points of the target object with the preset shape accounts for 70% of the total number of point cloud points in the above-mentioned point cloud data to be detected, which is obviously greater than the preset ratio. Therefore, it can be determined that the scene to be detected contains spherical or cylindrical target objects, that is, it contains abnormal parts.

[0098] In step S303, after the step of "confirming that the scene to be detected contains the target object of the preset shape", the target detection method described in the second embodiment may further include: issuing a prompt message to prompt that the scene to be detected contains the target object of the preset shape.

[0099] In addition, in the second embodiment of the present application, if the point cloud data to be detected in step S301 is a point cloud block after preprocessing and clustering the point cloud data of the scene to be detected, then after step S302, the target detection method described in the second embodiment may further include:

[0100] If the ratio of the number of point cloud points whose shape is the preset shape to the total number of point cloud points in the above-mentioned point cloud data to be detected is less than or equal to the above-mentioned preset ratio, then

[0101] The selected point cloud blocks are removed, and another point cloud block is selected from each point cloud block as the point cloud data to be detected, and the process returns to step S302 until all point cloud blocks are traversed.

[0102] In addition, in order to more accurately identify the package shape to which each point cloud point in the point cloud data to be detected belongs, in addition to the point cloud data to be detected, the grayscale value (for example, RGB value) corresponding to each point cloud point in the point cloud data to be detected can also be obtained; accordingly, the above-mentioned step S302 is specifically: the point cloud data to be detected and the grayscale value corresponding to each point cloud point in the point cloud data to be detected are input into the trained second neural network model, so that the second neural network model recognizes the shape of the target object to which each point cloud point in the point cloud data to be detected belongs (in this case, technicians in this field can easily understand that when training the second neural network model, it not only depends on the sample point cloud data to be detected, but also depends on the grayscale value of each point cloud point in the sample point cloud data to be detected).

[0103] It can be seen that the target detection method of the second embodiment uses a neural network model to identify the shape of the target object to which each point cloud point belongs, and then determines whether there is a target object of a preset shape in the scene to be detected. When the training samples are sufficient, the trained neural network model can accurately identify the shape of the target object to which each point cloud point belongs, thereby further accurately detecting whether there is a target object of a preset shape in the scene to be detected. Therefore, the target detection method can automatically identify the target object of a preset shape without relying on manual work, and can improve the recognition efficiency of the target object shape to a certain extent.

[0104] It should be understood that the size of the serial numbers of the steps in the above method embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0105] Embodiment 3

[0106] The following discusses a target detection device. For the sake of convenience, only the parts related to this application are shown. Figure 5 As shown, the target detection device 500 includes:

[0107] The to-be-detected point cloud acquisition module 501 is used to acquire to-be-detected point cloud data, where the to-be-detected point cloud data is point cloud data derived from a to-be-detected scene, where the to-be-detected scene includes a to-be-detected target;

[0108] The target surface detection module 502 is used to input the above-mentioned point cloud data to be detected into the trained first neural network model, so that the first neural network model can identify each point cloud point belonging to the target surface of the above-mentioned target to be detected in the above-mentioned point cloud data;

[0109] The surface size determination module 503 is used to determine the size of the target surface according to each point cloud point belonging to the target surface.

[0110] Optionally, the above-mentioned to-be-detected point cloud acquisition module 501 includes:

[0111] A scene point cloud acquisition unit, used to acquire point cloud data of the scene to be detected;

[0112] A preprocessing unit is used to preprocess the point cloud data of the scene to be detected to obtain the point cloud data to be detected, wherein the preprocessing includes downsampling, denoising and / or removing point cloud points in the point cloud data of the scene to be detected that are not the target to be detected.

[0113] Optionally, the scene point cloud acquisition unit includes:

[0114] A depth image acquisition subunit, used to acquire a depth image of the scene to be detected acquired by a depth image acquisition device;

[0115] The depth image conversion subunit is used to convert each pixel point in the depth image into each point cloud point to obtain the point cloud data of the scene to be detected, and the point cloud data of the scene to be detected is composed of the converted point cloud points.

[0116] Optionally, the image acquisition direction of the depth image acquisition device is perpendicular to a supporting plane supporting the target to be detected, the target surface is an upper surface of the target to be detected, and the three-dimensional coordinates corresponding to each point cloud point in the point cloud data to be detected are coordinates in a coordinate system with the image acquisition direction of the depth image acquisition device as the z-axis;

[0117] Accordingly, the surface size determination module 503 includes:

[0118] A projection unit, used for projecting each point cloud point belonging to the target surface onto an xoy two-dimensional plane;

[0119] A circumscribed rectangle unit is used to determine the minimum circumscribed rectangle of each point cloud point projected onto a two-dimensional plane;

[0120] The length and width determination unit is used to determine the length and width of the minimum circumscribed rectangle according to the two-dimensional coordinates of each point cloud point projected onto the xoy two-dimensional plane, so as to obtain the length and width of the target surface.

[0121] Optionally, the target detection device 500 further includes:

[0122] A first distance module, used to obtain a first distance value between the depth image acquisition device and the supporting plane;

[0123] A second distance module, used to determine a second distance value between the target plane and the depth image acquisition device according to the point cloud points located on the target surface;

[0124] The height determination module is used to determine the difference between the first distance value and the second distance value as the height of the target to be detected.

[0125] Optionally, the target detection device 500 further includes:

[0126] The volume determination module is used to calculate the volume of the target to be detected according to the length and width of the target surface and the height of the target to be detected.

[0127] It should be noted that the information interaction, execution process, etc. between the devices / units in the third embodiment are based on the same concept as the first embodiment of the present application. Their specific functions and technical effects can be found in part of the embodiment and will not be repeated here.

[0128] Embodiment 4

[0129] Another target detection device is discussed below. For the sake of convenience, only the parts related to the present application are shown. Figure 6 As shown, the target detection device 600 includes:

[0130] The to-be-detected point cloud acquisition module 601 is used to acquire to-be-detected point cloud data, where the to-be-detected point cloud data is derived from point cloud data of a to-be-detected scene, where the to-be-detected scene includes one or more target objects;

[0131] The shape detection module 602 is used to input the above-mentioned point cloud data to be detected into the trained second neural network model, so that the second neural network model recognizes the shape of the target object to which each point cloud point in the above-mentioned point cloud data to be detected belongs;

[0132] The preset shape determination module 603 is used to confirm that the scene to be detected contains a target object of the preset shape if the ratio of the number of point cloud points whose shape is the preset shape to the total number of point cloud points in the point cloud data to be detected is greater than the preset ratio.

[0133] Optionally, the above-mentioned to-be-detected point cloud acquisition module 601 includes:

[0134] A scene point cloud acquisition unit, used to acquire point cloud data of the scene to be detected;

[0135] A preprocessing unit is used to preprocess the point cloud data of the scene to be detected to obtain the point cloud data to be detected, wherein the preprocessing includes downsampling, denoising and / or removing point cloud points that do not belong to any target object in the point cloud data of the scene to be detected.

[0136] Optionally, the preprocessing unit includes:

[0137] A preprocessing subunit, used for preprocessing the point cloud data of the scene to be detected to obtain preprocessed point cloud data;

[0138] The clustering subunit is used to cluster the preprocessed point cloud data to obtain point cloud blocks, and select a point cloud block as the point cloud data to be detected.

[0139] Optionally, the target detection device 600 further includes:

[0140] A return module is used for, if the ratio of the number of point cloud points whose shape is the preset shape to the total number of point cloud points of the point cloud data to be detected is less than or equal to the preset ratio,

[0141] Remove the selected point cloud blocks, select another point cloud block from each point cloud block as the point cloud data to be detected, and trigger the shape detection module 602 to continue the step of inputting the point cloud data to be detected into the trained second neural network model so that the second neural network model recognizes the shape of the target object to which each point cloud point in the point cloud data to be detected belongs, until all point cloud blocks are traversed.

[0142] Optionally, the target detection device 600 further includes:

[0143] A grayscale acquisition module, used to acquire the grayscale value corresponding to each point cloud point in the point cloud data to be detected;

[0144] Accordingly, the shape detection module 602 is specifically used for:

[0145] The point cloud data to be detected and the grayscale value corresponding to each point cloud point in the point cloud data to be detected are input into the trained second neural network model, so that the second neural network model can recognize the shape of the target object to which each point cloud point in the point cloud data to be detected belongs.

[0146] Optionally, the target detection device 600 further includes:

[0147] The prompt module is used to issue a prompt message to prompt that the scene to be detected contains a target object of the preset shape.

[0148] It should be noted that the information interaction, execution process, etc. between the devices / units in this embodiment 4 are based on the same concept as those in embodiment 2 of the present application. Their specific functions and technical effects can be found in embodiment 2 and will not be repeated here.

[0149] Embodiment 5

[0150] Figure 7 Schematic diagram of the terminal device provided in Embodiment 5 of the present application. Figure 7 As shown, the terminal device 700 of this embodiment includes: a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 703, the steps in the above-mentioned method embodiments (embodiment 1 and embodiment 2) are implemented. Alternatively, when the processor 701 executes the computer program 703, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0151] Exemplarily, the computer program 703 may be divided into one or more modules / units, which are stored in the memory 702 and executed by the processor 701 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 703 in the terminal device 700. For example, the computer program 703 may be divided into a point cloud acquisition module to be detected, a target surface detection module, and a surface size determination module, and the specific functions of each module are as follows:

[0152] Acquire point cloud data to be detected, where the point cloud data to be detected is point cloud data derived from a scene to be detected, and the scene to be detected includes a target to be detected;

[0153] Inputting the above-mentioned point cloud data to be detected into the trained first neural network model, so that the first neural network model recognizes each point cloud point belonging to the target surface of the above-mentioned target to be detected in the above-mentioned point cloud data to be detected;

[0154] The size of the target surface is determined according to each point cloud point belonging to the target surface.

[0155] Alternatively, the computer program 703 may be divided into a to-be-detected point cloud acquisition module, a shape detection module, and a preset shape determination module, and the specific functions of each module are as follows:

[0156] Acquire point cloud data to be detected, where the point cloud data to be detected is derived from point cloud data of a scene to be detected, where the scene to be detected includes one or more target objects;

[0157] Inputting the above-mentioned point cloud data to be detected into the trained second neural network model, so that the second neural network model recognizes the shape of the target object to which each point cloud point in the above-mentioned point cloud data to be detected belongs;

[0158] If the ratio of the number of point cloud points whose shape is a preset shape belongs to the target object is greater than the preset ratio to the number of all point cloud points in the above-mentioned point cloud data to be detected, it is confirmed that the above-mentioned scene to be detected contains a target object of the above-mentioned preset shape.

[0159] The terminal device may include, but is not limited to, a processor 701 and a memory 702. Those skilled in the art will appreciate that Figure 7 It is only an example of the terminal device 700 and does not constitute a limitation of the terminal device 700. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0160] The processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0161] The memory 702 may be an internal storage unit of the terminal device 700, such as a hard disk or memory of the terminal device 700. The memory 702 may also be an external storage device of the terminal device 700, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 700. Further, the memory 702 may also include both an internal storage unit of the terminal device 700 and an external storage device. The memory 702 is used to store the computer program and other programs and data required by the terminal device. The memory 702 may also be used to temporarily store data that has been output or is to be output.

[0162] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0163] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0164] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0165] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the above modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0166] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0168] If the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned various method embodiments, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0169] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A target detection method, characterized in that: include: Acquire point cloud data to be detected, wherein the point cloud data to be detected is point cloud data derived from a scene to be detected, and the scene to be detected includes a target to be detected; The target to be detected is a package located on the conveyor belt surface; Inputting the point cloud data to be detected into a trained first neural network model so that the first neural network model recognizes each point cloud point in the point cloud data to be detected that belongs to the target surface of the target to be detected; The target surface is the upper surface of the target to be detected; Determining the size of the target surface according to each point cloud point belonging to the target surface; The step of obtaining the point cloud data to be detected includes: Acquire point cloud data of the scene to be detected; Preprocessing the point cloud data of the scene to be detected to obtain the point cloud data to be detected, wherein the preprocessing includes removing point cloud points that are not the target to be detected in the point cloud data of the scene to be detected; The training process of the first neural network includes: Obtain sample point cloud data to be detected; Manually marking a label of each point cloud point in the sample to be detected point cloud data, wherein the label includes a package upper surface, a package side surface, and a belt surface; Based on the sample point cloud data to be detected and the label corresponding to each point cloud point, a first neural network is trained, wherein the output of the first neural network is the surface to which each point cloud point belongs, including the upper surface of the package, the side surface of the package, or the belt surface; The method further comprises: Inputting the point cloud data to be detected into the trained second neural network model, so that the second neural network model can recognize the shape of the target object to which each point cloud point in the point cloud data to be detected belongs; the shape of the target object includes: normal shape, spherical shape and cylindrical shape; If the ratio of the number of point cloud points of the target object whose shape is a preset shape to the number of all point cloud points of the point cloud data to be detected is greater than a preset ratio, it is confirmed that the scene to be detected contains a target object of the preset shape; The preset shapes are spherical and cylindrical; After the step of inputting the point cloud data to be detected into the trained second neural network model so that the second neural network model recognizes the shape of the target object to which each point cloud point in the point cloud data to be detected belongs, the method further includes: If the ratio of the number of point cloud points whose shape is the preset shape to the number of all point cloud points in the point cloud data to be detected is less than or equal to the preset ratio, then Remove the selected point cloud blocks, select another point cloud block from each point cloud block as the point cloud data to be detected, and return to execute the step of inputting the point cloud data to be detected into the trained second neural network model so that the second neural network model recognizes the shape of the target object to which each point cloud point in the point cloud data to be detected belongs, and subsequent steps until all point cloud blocks are traversed; After the step of confirming that the scene to be detected contains the target object of the preset shape, the target detection method further includes: A prompt message is issued to prompt that the scene to be detected contains a target object of the preset shape.

2. The target detection method according to claim 1, characterized in that: The preprocessing also includes downsampling and denoising the point cloud data of the scene to be detected.

3. The target detection method according to claim 2, characterized in that: The step of obtaining the point cloud data of the scene to be detected includes: Acquire a depth image of the scene to be detected acquired by a depth image acquisition device; Each pixel point in the depth image is converted into each point cloud point to obtain the point cloud data of the scene to be detected, and the point cloud data of the scene to be detected is composed of each point cloud point obtained by the conversion.

4. The target detection method according to claim 3, characterized in that: The image acquisition direction of the depth image acquisition device is perpendicular to the supporting plane supporting the target to be detected, the target surface is the upper surface of the target to be detected, and the three-dimensional coordinates corresponding to each point cloud point in the point cloud data to be detected are coordinates in a coordinate system with the image acquisition direction of the depth image acquisition device as the z-axis; Accordingly, determining the size of the target surface according to each point cloud point belonging to the target surface includes: Projecting each point cloud point belonging to the target surface onto the xoy two-dimensional plane; Determine the minimum circumscribed rectangle of each point cloud point projected onto the two-dimensional plane; According to the two-dimensional coordinates of each point cloud point projected onto the xoy two-dimensional plane, the length and width of the minimum circumscribed rectangle are determined to obtain the length and width of the target surface.

5. The target detection method according to claim 4, characterized in that: The target detection method further includes: Acquire a first distance value between the depth image acquisition device and the supporting plane; Determining a second distance value between the target surface and the depth image acquisition device according to the point cloud points located on the target surface; The difference between the first distance value and the second distance value is determined as the height of the target to be detected.

6. The target detection method according to claim 5, characterized in that: The target detection method further includes: The volume of the target to be detected is calculated according to the length and width of the target surface and the height of the target to be detected.

7. The target detection method according to claim 1, characterized in that: The preprocessing of the point cloud data of the scene to be detected to obtain the point cloud data to be detected includes: Preprocessing the point cloud data of the scene to be detected to obtain preprocessed point cloud data; The preprocessed point cloud data are clustered to obtain point cloud blocks, and a point cloud block is selected as the point cloud data to be detected.

8. The target detection method according to any one of claims 1 to 7, characterized in that: Before the step of inputting the point cloud data to be detected into the trained second neural network model so that the second neural network model recognizes the shape of the target object to which each point cloud point in the point cloud data to be detected belongs, the target detection method further includes: Obtaining the grayscale value corresponding to each point cloud point in the point cloud data to be detected; Accordingly, the point cloud data to be detected is input into the trained second neural network model so that the second neural network model recognizes the shape of the target object to which each point cloud point in the point cloud data to be detected belongs, specifically: The point cloud data to be detected and the grayscale value corresponding to each point cloud point in the point cloud data to be detected are input into the trained second neural network model, so that the second neural network model can recognize the shape of the target object to which each point cloud point in the point cloud data to be detected belongs.

9. A target detection device, characterized in that: include: A point cloud acquisition module to be detected is used to acquire point cloud data to be detected, wherein the point cloud data to be detected is point cloud data derived from a scene to be detected, wherein the scene to be detected includes a target to be detected; the target to be detected is a package located on a conveyor belt surface; A target surface detection module, used for inputting the point cloud data to be detected into a trained first neural network model, so that the first neural network model identifies each point cloud point belonging to the target surface of the target to be detected in the point cloud data to be detected; the target surface is the upper surface of the target to be detected; A surface size determination module, used to determine the size of the target surface according to each point cloud point belonging to the target surface; The step of obtaining the point cloud data to be detected includes: Acquire point cloud data of the scene to be detected; Preprocessing the point cloud data of the scene to be detected to obtain the point cloud data to be detected, wherein the preprocessing includes removing point cloud points that are not the target to be detected in the point cloud data of the scene to be detected; The training process of the first neural network includes: Obtain sample point cloud data to be detected; Manually marking a label of each point cloud point in the sample to be detected point cloud data, wherein the label includes a package upper surface, a package side surface, and a belt surface; Based on the sample point cloud data to be detected and the label corresponding to each point cloud point, a first neural network is trained, wherein the output of the first neural network is the surface to which each point cloud point belongs, including the upper surface of the package, the side surface of the package, or the belt surface; The device also includes: A shape detection module, used for inputting the point cloud data to be detected into a trained second neural network model, so that the second neural network model recognizes the shape of the target object to which each point cloud point in the point cloud data to be detected belongs; the shape of the target object includes: a normal shape, a sphere and a cylinder; A preset shape determination module, configured to confirm that the scene to be detected contains a target object of the preset shape if the ratio of the number of point cloud points of the target object identified as having a preset shape to the number of all point cloud points of the point cloud data to be detected is greater than a preset ratio; The preset shapes are spherical and cylindrical; The target detection device further includes: A return module is used for, if the ratio of the number of point cloud points whose shape is the preset shape to the total number of point cloud points of the point cloud data to be detected is less than or equal to the preset ratio, Remove the selected point cloud blocks, select another point cloud block from each point cloud block as the point cloud data to be detected, and return to execute the step of inputting the point cloud data to be detected into the trained second neural network model so that the second neural network model recognizes the shape of the target object to which each point cloud point in the point cloud data to be detected belongs, and subsequent steps until all point cloud blocks are traversed; The prompt module is used to issue a prompt message to prompt that the scene to be detected contains a target object of the preset shape.

10. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images