An image detection method and apparatus

Abstract

The application discloses an image detection method and device, and relates to the technical field of computer vision. A specific embodiment of the method comprises the following steps: performing instance segmentation on a to-be-detected image by using an image instance segmentation model to obtain each instance in the to-be-detected image; calculating preliminary centers of the each instance based on point cloud data and the each instance; correcting the preliminary centers of the each instance by using an instance center correction model to obtain corrected centers of the each instance; and inputting the corrected centers of the each instance into a target detection model to output frame bodies and categories of the each instance. The embodiment can solve the technical problem of poor image detection performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to an image detection method and device. BACKGROUND

[0002] An autonomous vehicle is usually equipped with multiple types of sensors for detecting obstacles, such as a laser radar, an RGB image camera, a millimeter wave radar, etc. However, due to different imaging principles and installation angles of different sensors, the output results obtained are also different: for example, a multi-line laser radar installed on the top has an output of 360-degree point cloud, while an image sensor is composed of several RGB cameras, each of which has a fixed field of view angle and is responsible for generating an RGB image of the corresponding partial region. Due to the different data modalities generated by these sensors, the characteristics of the data generated are also different: for example, a laser radar is not sensitive to light intensity and can be used at night, but due to the high cost of laser radars, mass-produced autonomous vehicles generally cannot use high-line beam products, so the obtained point cloud data is sparse, resulting in low resolution; while an RGB image camera has low cost, high resolution, and obvious texture features, but is easily affected by weather changes and environmental light intensity.

[0003] In the process of implementing the present application, the inventors have found that at least the following problems exist in the prior art:

[0004] From the algorithm aspect, usually a dedicated algorithm is used to obtain detection results for each modality of data, and then the detection results of each model are fused, which is generally referred to as a post-fusion method. In contrast, a pre-fusion method is used, that is, a deep model or algorithm is used to process all different modalities of data at one time to directly obtain the final detection results, such as the AVOD algorithm which uses a deep neural network to extract features from different modalities of data to obtain feature maps, and then fuses the feature maps at the feature map level. Due to the differences between different modalities of data, it is difficult to align the feature maps from different modalities, such as two feature maps from a 360-degree point cloud and a front-view camera. However, whether it is a post-fusion method or a pre-fusion method, the fusion effect is not good, resulting in poor image detection performance. SUMMARY

[0005] Therefore, the embodiments of the present application provide an image detection method and device to solve the technical problem of poor image detection performance.

[0006] To achieve the above object, according to one aspect of the embodiments of the present application, an image detection method is provided, comprising:

[0007] performing instance segmentation on the to-be-detected image using an image instance segmentation model to obtain each instance in the to-be-detected image;

[0008] calculating a preliminary center of each instance based on the point cloud data and the instances;

[0009] correcting the preliminary center of each instance by using an instance center correction model to obtain a corrected center of each instance;

[0010] inputting the corrected center of each instance into a target detection model to output a bounding box and a category of each instance.

[0011] Optionally, the calculating of the preliminary center of each instance based on the point cloud data and the instances comprises:

[0012] projecting the point cloud data to the instances;

[0013] merging the point cloud data of a same instance appearing in different images to be detected according to the point cloud data of an overlapping part of the instances;

[0014] calculating the point cloud data of a main body of each instance by using a clustering algorithm;

[0015] calculating the preliminary center of each instance based on the point cloud data of the main body of each instance.

[0016] Optionally, the calculating of the point cloud data of the main body of each instance by using a clustering algorithm comprises:

[0017] calculating the point cloud data of the main body of each instance by using a spatial clustering algorithm and removing the point cloud data other than the main body of each instance.

[0018] Optionally, before the instance segmentation of the image to be detected by using the image instance segmentation model to obtain each instance in the image to be detected, the method further comprises:

[0019] obtaining sample images at different angles, marking the positions, categories and outlines of sample instances on the sample images at the different angles, and training a first model to obtain an image instance segmentation model;

[0020] calculating preliminary centers of the sample instances, marking bounding boxes of the sample instances in sample point cloud data, and training a second model to obtain an instance center correction model;

[0021] training a third model to obtain a target detection model.

[0022] Optionally, the training of the first model to obtain the image instance segmentation model comprises:

[0023] inputting the sample images of the respective angles, positions, categories and contours of the respective sample instances into a first model for training, so as to obtain an image instance segmentation model through training;

[0024] The first model is Mask R-CNN, Hybrid Task Cascade or BlendMaskd.

[0025] Optionally, a preliminary center of each sample instance is calculated, a bounding box of each sample instance is marked in sample point cloud data, and a second model is used to obtain an instance center correction model, including:

[0026] calculating a preliminary center of each sample instance based on sample point cloud data and the respective sample instances;

[0027] marking a bounding box of each sample instance in the sample point cloud data, and calculating a center of the bounding box of each sample instance;

[0028] inputting the preliminary center of each sample instance and the center of the bounding box of each sample instance into a second model for training, so as to obtain an instance center correction model through training.

[0029] Optionally, the preliminary center of each sample instance is calculated based on sample point cloud data and the respective sample instances, including:

[0030] projecting sample point cloud data to the respective sample instances;

[0031] merging point cloud data of a same sample instance appearing in different sample images according to point cloud data of an overlapping part in the respective sample instances;

[0032] calculating point cloud data of a main body of each sample instance by using a clustering algorithm;

[0033] calculating a preliminary center of each sample instance based on the point cloud data of the main body of each sample instance.

[0034] Optionally, the second model is PointNet, PointNet++, Cylider3D or Rand-LA.

[0035] Optionally, a third model is used to obtain a target detection model, including:

[0036] inputting attribute data of each point cloud of the respective sample instances and the bounding box of each sample instance into a third model for training, so as to obtain a target detection model through training;

[0037] The attribute data of each point cloud includes three-dimensional coordinates, a category, and a center coordinate of a bounding box.

[0038] Optionally, the third model is PointPillars, VoxelNet, Sparsely Embedded Convolutional Detection, or CenterPoint.

[0039] In addition, according to another aspect of an embodiment of the present application, an image detection device is provided, comprising:

[0040] a segmentation module configured to perform instance segmentation on the to-be-detected image by using an image instance segmentation model to obtain each instance in the to-be-detected image;

[0041] a calculation module configured to calculate a preliminary center of each instance based on point cloud data and the instances;

[0042] a correction module configured to correct the preliminary center of each instance by using an instance center correction model to obtain a corrected center of each instance;

[0043] a detection module configured to input the corrected center of each instance into a target detection model to output a bounding box and a category of each instance.

[0044] Optionally, the calculation module is further configured to:

[0045] project the point cloud data onto the instances;

[0046] merge point cloud data of a same instance appearing in different to-be-detected images according to point cloud data of an overlapping part of the instances;

[0047] calculate point cloud data of a main body of each instance by using a clustering algorithm;

[0048] calculate a preliminary center of each instance based on the point cloud data of the main body of each instance.

[0049] Optionally, the calculation module is further configured to:

[0050] calculate point cloud data of a main body of each instance by using a spatial clustering algorithm, and remove point cloud data other than the main body of each instance.

[0051] Optionally, the device further comprises a training module configured to:

[0052] obtain sample images at various angles, mark positions, categories, and outlines of various sample instances on the sample images at the various angles, and train an image instance segmentation model by using a first model;

[0053] calculate a preliminary center of each sample instance, mark a bounding box of each sample instance in sample point cloud data, and train a second model to obtain an instance center correction model;

[0054] train a third model to obtain a target detection model.

[0055] Optionally, the training module is further configured to:

[0056] input the sample images of each angle and the position, category and contour of each sample instance of the sample images into a first model for training, so as to train an image instance segmentation model;

[0057] The first model is Mask R-CNN, Hybrid Task Cascade or BlendMaskd.

[0058] Optionally, the training module is further configured to:

[0059] calculate a preliminary center of each sample instance based on the sample point cloud data and the sample instances;

[0060] mark a bounding box of each sample instance in the sample point cloud data, and calculate a center of the bounding box of each sample instance;

[0061] input the preliminary center of each sample instance and the center of the bounding box of each sample instance into a second model for training, so as to train an instance center correction model.

[0062] Optionally, the training module is further configured to:

[0063] project sample point cloud data to each sample instance;

[0064] merge point cloud data of a same sample instance appearing in different sample images according to point cloud data of an overlapping part in each sample instance;

[0065] calculate point cloud data of a main body of each sample instance by using a clustering algorithm;

[0066] calculate a preliminary center of each sample instance based on the point cloud data of the main body of each sample instance.

[0067] Optionally, the second model is PointNet, PointNet++, Cylider3D or Rand-LA.

[0068] Optionally, the training module is further configured to:

[0069] input attribute data of each point cloud of the respective sample instances and the bounding boxes of the respective sample instances into a third model for training, so as to obtain a target detection model.

[0070] The attribute data of each point cloud comprises three-dimensional coordinates, a category, and a center coordinate of the bounding box to which the point cloud belongs.

[0071] Optionally, the third model is PointPillars, VoxelNet, Sparsely EmbeddedConvolutional Detection, or CenterPoint.

[0072] According to another aspect of the embodiments of the present application, an electronic device is also provided, comprising:

[0073] one or more processors;

[0074] a storage device configured to store one or more programs,

[0075] When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of the above embodiments.

[0076] According to another aspect of the embodiments of the present application, a computer readable medium having a computer program stored thereon is also provided, the program being executed by a processor to implement the method according to any one of the above embodiments.

[0077] One embodiment of the above application has the following advantages or beneficial effects: because the image instance segmentation model is used to perform instance segmentation on the image to be detected, the instance center correction model is used to correct the preliminary center of each instance, and the corrected center of each instance is input into the target detection model, the technical means overcome the technical problem of poor image detection performance in the prior art. The instance segmentation result of the image is used to label each point in the point cloud at the instance level, and the center of each instance is further corrected, and then the point cloud detection algorithm is used to perform target detection. Since the instance information of the point cloud is obtained before the point cloud detection algorithm is input, the learning difficulty of the detection algorithm is greatly reduced, thereby greatly improving the detection performance.

[0078] The further effects of the above non-conventional optional mode will be described in the following combined with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0079] The accompanying drawings are used to better understand the present application, and do not constitute an improper limitation on the present application. Among them:

[0080] Figure 1is a schematic diagram of the main flow of the image detection method according to an embodiment of the present application;

[0081] Figure 2 is a schematic diagram of the main flow of the image detection method according to a reference embodiment of the present application;

[0082] Figure 3 is a schematic diagram of the main modules of the image detection device according to an embodiment of the present application;

[0083] Figure 4 is an exemplary system architecture diagram to which embodiments of the present application can be applied;

[0084] Figure 5 is a structural schematic diagram of a computer system of a terminal device or a server suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0085] Exemplary embodiments of the present application are described below with reference to the accompanying drawings, which include various details of the embodiments of the present application to assist in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0086] Embodiments of the present application use instance segmentation results of images to label each point in the point cloud at the instance level. After instance segmentation in the first stage, each point cloud data has a class label, such as car, pedestrian, bicycle, etc., and different instances of the same class can also be identified, such as 10 cars, 20 pedestrians, etc. Instance segmentation can distinguish these instances of the same class, such as car 1, car 2, car 3, etc.

[0087] After projecting the point cloud data to the result of the instance segmentation model, embodiments of the present application further use the image instance segmentation model to obtain independent information for each instance, such as calculating the center of each point cloud data corresponding to the instance and correcting it. Therefore, using this information in the second stage input data can further reduce the difficulty of algorithm learning, thereby improving the accuracy of image detection. Therefore, embodiments of the present application make full use of the information in the image, reduce the difficulty of point cloud detection in the second stage, and thus improve the detection performance.

[0088] The embodiment of the present application breaks the absolute "barrier" of post-fusion and pre-fusion, and ingeniously uses the result of image instance segmentation, and then projects the original point cloud data onto the result of image instance segmentation according to the calibration relationship between the point cloud and the camera, so that each point cloud data obtains entity classification such as car, pedestrian, bicycle, etc., and then the center of each instance in the radar coordinate system is calculated, and then the traditional point cloud detection algorithm is used for target detection. Since the instance information of the point cloud is obtained before being input into the point cloud detection algorithm, the learning difficulty of the detection algorithm is greatly reduced, thereby greatly improving the detection performance. The embodiment of the present application is different from the post-fusion method, which does not fuse the detection results of two modalities, and is different from the pre-fusion method, because it does not directly process the data of two modalities at the same time, but processes them sequentially. Due to the sequential relationship between the two, it can be regarded as a two-stage image-point cloud detection method.

[0089] Figure 1 is a schematic diagram of the main process of the image detection method according to the embodiment of the present application. As shown in Figure 1 , the image detection method can include:

[0090] Step 101, using an image instance segmentation model to perform instance segmentation on the to-be-detected image to obtain each instance in the to-be-detected image.

[0091] In the inference stage, a frame of point cloud data and at least one corresponding RGB image (i.e. the to-be-detected image, which can be collected by RGB cameras installed at different angles) are needed. It should be noted that the to-be-detected image can be one or multiple, which is determined by the number of RGB cameras. If multiple cameras are installed at different angles, one to-be-detected image is collected by each camera, and then instance segmentation is performed on each to-be-detected image. If only one camera is installed, instance segmentation is performed on the image collected by the camera.

[0092] In the embodiment of the present application, first, an image instance segmentation model is used to perform instance segmentation on the to-be-detected image at each angle to obtain each instance in each to-be-detected image. The image instance segmentation model identifies which instance each pixel belongs to, such as which pixels belong to the first car and which pixels belong to the second car.

[0093] In order to obtain a better instance segmentation result, the image instance segmentation model needs to be pre-trained. Optionally, before step 101, the method further comprises: obtaining sample images at various angles, marking the positions, categories and contours of various sample instances on the sample images at various angles, and training the first model to obtain an image instance segmentation model; calculating preliminary centers of the various sample instances, marking the frame bodies of the various sample instances in sample point cloud data, and training the second model to obtain an instance center correction model; and training the third model to obtain a target detection model. Preferably, the sample images at various angles are collected by RGB cameras installed at various angles, the labels of the instances to which each pixel in the sample images belong are marked, and the image instance segmentation model is trained based on the marking result. Then, the preliminary centers of the various sample instances are calculated, and the instance center correction model is trained in combination with the frame bodies of the various sample instances marked in the sample point cloud data. Finally, the target detection model is trained. In the embodiment of the application, the image instance segmentation model, the instance center correction model and the target detection model need to be pre-trained, so that accurate target detection can be performed in the subsequent detection step.

[0094] Optionally, the first model is used to train the image instance segmentation model, which comprises: inputting the sample images at various angles and the positions, categories and contours of the various sample instances into the first model for training, so as to train the image instance segmentation model. Preferably, the sample images at various angles are collected by RGB cameras installed at various angles, and then the labels of the instances to which each pixel in the sample images belong are marked. Then, the sample images at various angles and the marking result are inputted into the first model for training, and the image instance segmentation model is obtained through iterative training. Optionally, the first model is Mask R-CNN, Hybrid Task Cascade or BlendMaskd. Through training of the sample images at various angles by using these models, a better training result can be obtained.

[0095] Optionally, the preliminary center of each sample instance is calculated, the bounding box of each sample instance is marked in the sample point cloud data, and the instance center correction model is obtained by training the second model, which can include: calculating the preliminary center of each sample instance based on the sample point cloud data and the sample instance; marking the bounding box of each sample instance in the sample point cloud data and calculating the center of the bounding box of each sample instance; inputting the preliminary center of each sample instance and the center of the bounding box of each sample instance into the second model for training, thereby obtaining the instance center correction model. In order to make full use of the image information obtained in step 101 for point cloud detection in subsequent steps, the instance center correction model needs to be trained in advance. When training the instance center correction model, the preliminary center of each sample instance is used as input data, the center of the bounding box of each sample instance in the labeled data is used as a regression target, and the two parts of data are input into the second model for training. Through iterative training, the instance center correction module is obtained. The module is used to correct the instance center, so that the learning difficulty of the detection algorithm is greatly reduced, thereby greatly improving the detection performance.

[0096] Specifically, the whole point cloud data is projected onto the corresponding RGB image, the instance of the point cloud is obtained in combination with the result of image instance segmentation, and then the preliminary center [c1, c2, c3] of each instance is calculated. At the same time, the bounding box of each sample instance is marked in the sample point cloud data, so as to calculate the center of the bounding box of each sample instance. The format of the point cloud data input into the second model is [x, y, z, r, c1, c2, c3], and the corresponding true value is [x, y, z, r, c1 * , c2 * , c3 * ]. The format of the center of the bounding box of each sample instance input into the second model is [x, y, z, r, c1', c2', c3'], which is used as a regression target, wherein [c1', c2', c3'] is the center of the bounding box of the marked sample instance.

[0097] Optionally, the preliminary center of each sample instance is calculated based on the sample point cloud data and the sample instance, which includes: projecting the sample point cloud data onto each sample instance; merging the point cloud data of the same sample instance appearing in different sample images according to the point cloud data of the overlapping part in each sample instance; calculating the point cloud data of the main body of each sample instance by using a clustering algorithm; and calculating the preliminary center of each sample instance based on the point cloud data of the main body of each sample instance.

[0098] First, the sample point cloud data is projected to each sample instance, and then the point cloud data of the overlapping part is merged into the point cloud data corresponding to the same sample instance according to the point cloud data of the overlapping part in each sample instance, and then the point cloud data of the main body of each sample instance is calculated by using a clustering algorithm, and finally the preliminary center of each sample instance is calculated. It should be pointed out that the same sample instance may appear in multiple sample images at different angles, so there will be overlapping point clouds in multiple sample images at different angles, and the point cloud data of the overlapping part needs to be merged into the point cloud data corresponding to the same sample instance, and then the preliminary center of each sample instance is calculated.

[0099] Optionally, after the point cloud data of the main body of each sample instance is calculated by using the clustering algorithm, the ground point cloud data needs to be removed to avoid the influence of the ground point cloud data on the clustering of the main body point cloud data.

[0100] Optionally, calculating the point cloud data of the main body of each sample instance by using the clustering algorithm comprises: calculating the point cloud data of the main body of each sample instance by using a spatial clustering algorithm, and removing the point cloud data other than the main body of each instance. The point cloud data can be clustered by using the clustering algorithm, and each cluster represents a sample instance, so that the main body of each sample instance can be accurately found, and the point cloud data other than the main body of each instance can be removed.

[0101] Optionally, the clustering algorithm can be DBSCAN, FN-DBSCAN, landmark FN-DBSCAN or OPTICS, and these clustering algorithms can accurately find the main body of each sample instance.

[0102] Optionally, the second model is PointNet, PointNet++, Cylider3D or Rand-LA, and these models can be used to train an instance center correction model, so as to accurately correct the instance center, greatly reduce the learning difficulty of the point cloud detection algorithm, and greatly improve the point cloud detection performance.

[0103] It should be pointed out that accurately calculating the center of each sample instance can further reduce the learning difficulty of the point cloud detection algorithm, but the result of the projection of the point cloud to the image has a certain deviation due to the influence of the accuracy of the laser radar-camera calibration and the time difference between the frames of different sensors, and finally the projection result is not accurate, especially at the edge of the instance segmentation. Therefore, the embodiment of the application uses a series of projection point cloud post-processing and model correction methods to obtain a better instance center.

[0104] Optionally, the target detection model is trained by using the third model, which can include: inputting the attribute data of each point cloud of each sample instance and the bounding box of each sample instance into the third model for training, so as to train the target detection model; wherein the attribute data of each point cloud includes three-dimensional coordinates, a category and the center coordinates of the bounding box. In the process of training the target detection model, the attribute data of each dimension of each point cloud needs to be input into the third model, for example, the format of the point cloud input into the third model is [x, y, z, r, cls, c1', c2', c3'], wherein x, y and z are three-dimensional coordinates, r is reflectivity, cls is the category corresponding to the image instance segmentation, and c1', c2' and c3' are the center coordinates of the bounding box to which the point cloud belongs.

[0105] Optionally, the third model is PointPillars, VoxelNet, Sparsely EmbeddedConvolutional Detection or CenterPoint, and the target detection model trained by using these models can improve the accuracy of image detection.

[0106] In step 102, the preliminary center of each instance is calculated based on the point cloud data and the instances.

[0107] In the step, the whole field point cloud data collected by the laser radar is projected to each instance, so as to calculate the preliminary center [c1, c2, c3] of each instance. It should be pointed out that the embodiment of the application calculates the preliminary center of each instance in the radar coordinate system, and the preliminary center can also be calculated in the image coordinate system, as long as the training stage and the inference stage are in the same coordinate system, and the embodiment of the application does not limit the coordinate system.

[0108] Optionally, step 102 can include: projecting the point cloud data to the instances; merging the point cloud data of the same instance appearing in different images to be detected according to the point cloud data of the overlapping part in the instances; calculating the point cloud data of the main body of the instances by using a clustering algorithm; and calculating the preliminary center of the instances based on the point cloud data of the main body of the instances. Similar to the training stage, first, the whole field point cloud data is projected to each instance, then the point cloud data of the overlapping part is merged into the point cloud data corresponding to the same sample instance according to the point cloud data of the overlapping part in each instance, then the point cloud data of the main body of each instance is calculated by using a clustering algorithm, and finally the preliminary center of each sample instance is calculated. It should be pointed out that since the same instance can appear in multiple images at different angles, there will be overlapping point clouds in multiple images at different angles, so the point cloud data of the overlapping part needs to be merged into the point cloud data corresponding to the same instance, and then the preliminary center of each instance is calculated.

[0109] Optionally, after the point cloud data of the main body of each instance is calculated by using the clustering algorithm, the ground point cloud data needs to be removed to avoid the influence of the ground point cloud data on the clustering of the main body point cloud data. Optionally, the point cloud data of the main body of each instance is calculated by using a spatial clustering algorithm, and the point cloud data outside the main body of each instance is removed. Optionally, the clustering algorithm can be DBSCAN, FN-DBSCAN, landmark FN-DBSCAN, or OPTICS, and these clustering algorithms can accurately find the main body of each instance.

[0110] In step 103, the preliminary center of each instance is corrected by using an instance center correction model to obtain the corrected center of each instance.

[0111] After the preliminary center [c1, c2, c3] of each instance is calculated, the preliminary center of each instance is corrected by using an instance center correction module. The output of the instance center correction module is [x, y, z, r, c1', c2', c3'], wherein [c1', c2', c3'] is the corrected instance center.

[0112] In step 104, the corrected center of each instance is input into a target detection model to output the bounding box and the category of each instance.

[0113] All points in the whole point cloud data are input into the target detection model in the format of [x, y, z, r, c1', c2', c3'] for each point. The target detection model outputs the bounding box and the category of each instance, that is, the bounding box and the category of the measured object.

[0114] According to the various embodiments described above, it can be seen that the embodiments of the present application solve the technical problem of poor image detection performance in the prior art by using an image instance segmentation model to perform instance segmentation on the image to be detected, using an instance center correction model to correct the preliminary center of each instance, and using the corrected center of each instance as the technical means of inputting the target detection model. The embodiments of the present application use the instance segmentation result of the image to perform instance-level labeling on each point in the point cloud, and further correct the center of each instance, and then perform target detection by using the point cloud detection algorithm. Since the instance information of the point cloud is obtained before being input into the point cloud detection algorithm, the learning difficulty of the detection algorithm is greatly reduced, thereby greatly improving the detection performance.

[0115] Figure 2 is a schematic diagram of the main process of the image detection method according to an embodiment of the present application. As another embodiment of the present application, as shown in Figure 2As shown, the image detection method can include:

[0116] In step 201, sample images at various angles are acquired, the positions, categories and contours of various sample instances are marked on the sample images at various angles, and a first model is trained to obtain an image instance segmentation model.

[0117] A plurality of frames of sample point cloud data and an RGB image corresponding to each frame of sample point cloud data can be acquired, and each frame of sample point cloud data can correspond to a plurality of RGB images (i.e., sample images, which can be acquired by RGB cameras installed at different angles).

[0118] Specifically, the label of the instance to which each pixel in the sample image belongs is marked, and an image instance segmentation model is trained based on the marking result. Optionally, the image instance segmentation model is trained by using the first model, including: inputting the sample images at various angles and the positions, categories and contours of various sample instances into the first model for training, so as to train the image instance segmentation model. Preferably, the sample images at various angles are acquired by RGB cameras installed at various angles, then the label of the instance to which each pixel in the sample image belongs is marked, and then the sample images at various angles and the marking result are input into the first model for training, so as to train the image instance segmentation model by iteration. Optionally, the first model is Mask R-CNN, Hybrid Task Cascade or BlendMaskd, and the sample images at various angles are trained by using these models, so that a better training result can be obtained.

[0119] In step 202, the preliminary centers of the various sample instances are calculated, the bounding boxes of the various sample instances are marked in the sample point cloud data, and a second model is trained to obtain an instance center correction model.

[0120] The preliminary centers of the various sample instances are calculated, and the bounding boxes of the various sample instances marked in the sample point cloud data are combined to train an instance center correction model. When the instance center correction model is trained, the preliminary centers of the various sample instances are taken as input data, the centers of the bounding boxes of the various sample instances in the labeling data are taken as regression targets, and the two parts of data are input into the second model for training, so as to obtain the instance center correction model by iteration. The instance center correction model is used to correct the instance center, so that the learning difficulty of the detection algorithm is greatly reduced, thereby greatly improving the detection performance. The format of the point cloud data input into the second model is [x, y, z, r, c1, c2, c3], the format of the center of the bounding box of each sample instance input into the second model is [x, y, z, r, c1', c2', c3'], and the regression target is [c1', c2', c3'], which is the center of the bounding box of the sample instance marked.

[0121] Step 203, a target detection model is obtained by training the third model.

[0122] Specifically, the attribute data of each point cloud of each sample instance and the bounding box of each sample instance are input into the third model for training, so as to obtain the target detection model; wherein the attribute data of each point cloud includes three-dimensional coordinates, a category and the center coordinates of the bounding box. In the process of training the target detection model, the attribute data of each dimension of each point cloud needs to be input into the third model, for example, the format of the point cloud input into the third model is [x, y, z, r, cls, c1', c2', c3'], wherein x, y and z are three-dimensional coordinates, r is reflectivity, cls is the category corresponding to the image instance segmentation, and c1', c2' and c3' are the center coordinates of the bounding box to which the point cloud belongs.

[0123] Step 204, performing instance segmentation on each to-be-detected image by using the image instance segmentation model to obtain each instance in each to-be-detected image.

[0124] In the inference stage, a frame of point cloud data and at least one corresponding RGB image (i.e. a to-be-detected image, which can be collected by RGB cameras installed at different angles) need to be obtained.

[0125] Step 205, calculating the preliminary center of each instance based on the point cloud data and each instance.

[0126] The whole field point cloud data collected by the laser lightning is projected to each instance to calculate the preliminary center [c1, c2, c3] of each instance. Specifically, the point cloud data is projected to the instances; the point cloud data of the same instance appearing in different images to be detected is combined according to the point cloud data of the overlapping part of the instances; the point cloud data of the main body of the instances is calculated by using a clustering algorithm; and the preliminary center of the instances is calculated based on the point cloud data of the main body of the instances. Similar to the training stage, the whole field point cloud data is first projected to the instances, then the point cloud data of the overlapping part is combined into the point cloud data corresponding to the same sample instance according to the point cloud data of the overlapping part of the instances, then the point cloud data of the main body of the instances is calculated by using a clustering algorithm, and finally the preliminary center of each sample instance is calculated. It should be noted that since the same instance can appear in multiple images at different angles, there will be overlapping point clouds in multiple images at different angles, so the point cloud data of the overlapping part needs to be combined into the point cloud data corresponding to the same instance, and then the preliminary center of each instance is calculated. Optionally, after the point cloud data of the main body of the instances is calculated by using the clustering algorithm, the ground point cloud data needs to be removed to avoid the influence of the ground point cloud data on the clustering of the main body point cloud data. Optionally, calculating the point cloud data of the main body of the instances by using the clustering algorithm includes: calculating the point cloud data of the main body of the instances by using a spatial clustering algorithm, and removing the point cloud data other than the main body of the instances. Optionally, the clustering algorithm can be DBSCAN, FN-DBSCAN, landmark FN-DBSCAN or OPTICS, etc. These clustering algorithms can accurately find the main body of each instance.

[0127] In step 206, the preliminary center of each instance is corrected by using an instance center correction model to obtain the corrected center of each instance.

[0128] After the preliminary center [c1, c2, c3] of each instance is calculated, the preliminary center of each instance is corrected by using an instance center correction module. The output of the instance center correction module is [x, y, z, r, c1', c2', c3'], wherein [c1', c2', c3'] is the corrected instance center.

[0129] In step 207, the corrected center of each instance is input into a target detection model to output the bounding box and the category of each instance.

[0130] All points in the whole field point cloud data are input into the target detection model in the format of [x, y, z, r, c1', c2', c3'] for each point. The target detection model outputs the bounding box and the category of each instance, that is, the bounding box and the category of the measured object.

[0131] In this embodiment, steps 201-203 are training stages, and steps 204-207 are inference stages.

[0132] In addition, the specific implementation of the image detection method in one of the embodiments of the present application has been described in detail in the above-described image detection method, and therefore repeated content will not be described here.

[0133] Figure 3 is a schematic diagram of the main modules of the image detection device according to an embodiment of the present application. As shown in Figure 3 The image detection device 300 includes a segmentation module 301, a calculation module 302, a correction module 303, and a detection module 304. The segmentation module 301 is configured to perform instance segmentation on a to-be-detected image using an image instance segmentation model to obtain each instance in the to-be-detected image. The calculation module 302 is configured to calculate a preliminary center of each instance based on point cloud data and the instances. The correction module 303 is configured to correct the preliminary center of each instance using an instance center correction model to obtain a corrected center of each instance. The detection module 304 is configured to input the corrected center of each instance into a target detection model to output a bounding box and a class of each instance.

[0134] Optionally, the calculation module 302 is further configured to:

[0135] project the point cloud data to the instances;

[0136] merge the point cloud data of the same instance appearing in different to-be-detected images according to the point cloud data of the overlapping part in the instances;

[0137] calculate the point cloud data of the main body of each instance using a clustering algorithm;

[0138] calculate the preliminary center of each instance based on the point cloud data of the main body of each instance.

[0139] Optionally, the calculation module 302 is further configured to:

[0140] calculate the point cloud data of the main body of each instance using a spatial clustering algorithm, and remove the point cloud data other than the main body of each instance.

[0141] Optionally, the image detection device further includes a training module configured to:

[0142] obtain sample images at various angles, mark the positions, classes, and contours of sample instances on the sample images at the various angles, and train a first model to obtain an image instance segmentation model;

[0143] calculate a preliminary center of each sample instance, mark a bounding box of each sample instance in sample point cloud data, and train a second model to obtain an instance center correction model;

[0144] train a third model to obtain a target detection model.

[0145] Optionally, the training module is further configured to:

[0146] input the sample images of each angle, the positions, categories and contours of the sample instances of each angle into a first model for training, so as to train an image instance segmentation model;

[0147] The first model is Mask R-CNN, Hybrid Task Cascade or BlendMaskd.

[0148] Optionally, the training module is further configured to:

[0149] calculate a preliminary center of each sample instance based on the sample point cloud data and the sample instances;

[0150] mark a bounding box of each sample instance in the sample point cloud data, and calculate a center of the bounding box of each sample instance;

[0151] input the preliminary center of each sample instance and the center of the bounding box of each sample instance into a second model for training, so as to train an instance center correction model.

[0152] Optionally, the training module is further configured to:

[0153] project sample point cloud data to each sample instance;

[0154] merge point cloud data of a same sample instance appearing in different sample images according to point cloud data of an overlapping part in each sample instance;

[0155] calculate point cloud data of a main body of each sample instance by using a clustering algorithm;

[0156] calculate a preliminary center of each sample instance based on the point cloud data of the main body of each sample instance.

[0157] Optionally, the second model is PointNet, PointNet++, Cylider3D or Rand-LA.

[0158] Optionally, the training module is further configured to:

[0159] The attribute data of each point cloud of each sample instance and the bounding box of each sample instance are input into a third model for training, so as to obtain a target detection model.

[0160] The attribute data of each point cloud includes three-dimensional coordinates, a category, and a center coordinate of the bounding box.

[0161] Optionally, the third model is PointPillars, VoxelNet, Sparsely EmbeddedConvolutional Detection, or CenterPoint.

[0162] It should be noted that the specific implementation of the image detection device is described in detail in the above image detection method, and thus the repeated content will not be described here.

[0163] Figure 4 An exemplary system architecture 400 to which embodiments of the present application can be applied is shown.

[0164] As shown in Figure 4 The system architecture 400 can include terminal devices 401, 402, 403, a network 404, and a server 405. The network 404 is used as a medium to provide a communication link between the terminal devices 401, 402, 403 and the server 405. The network 404 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0165] A user can use the terminal devices 401, 402, 403 to interact with the server 405 through the network 404 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 401, 402, 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0166] The terminal devices 401, 402, 403 can be various electronic devices with display screens and support for web browsing, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers, etc.

[0167] The server 405 can be a server that provides various services, such as a background management server that provides support for shopping websites browsed by users using the terminal devices 401, 402, 403 (only as an example). The background management server can analyze and process received item information query requests and other data, and feed back the processing results to the terminal devices.

[0168] It should be noted that the image detection method provided by the embodiment of the present application is generally executed by the server 405, and accordingly, the image detection apparatus is generally arranged in the server 405. The image detection method provided by the embodiment of the present application can also be executed by the terminal device 401, 402 or 403, and accordingly, the image detection apparatus can be arranged in the terminal device 401, 402 or 403.

[0169] It should be understood that Figure 4 The number of terminal devices, networks and servers in the system is merely illustrative. Any number of terminal devices, networks and servers can be provided according to the needs of implementation.

[0170] Reference is made below to Figure 5 which shows a structural schematic diagram of a computer system 500 of a terminal device suitable for use in implementing the embodiment of the present application. Figure 5 The terminal device shown is merely an example and should not impose any limitation on the functions and use range of the embodiment of the present application.

[0171] As shown in Figure 5 , the computer system 500 includes a central processing unit (CPU) 501 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 502 or loaded from a storage portion 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the system 500 are also stored in the RAM 503. The CPU 501, the ROM 502 and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0172] The following components are connected to the I / O interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 510 as needed, so that a computer program read therefrom is installed in the storage portion 508 as needed.

[0173] In particular, the processes described above with reference to the flow charts can be implemented as a computer software program in accordance with the embodiments disclosed herein. For example, embodiments disclosed herein include a computer program comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 509 and / or installed from the removable media 511. When the computer program is executed by the central processing unit (CPU) 501, the above-described functions defined in the system of the present application are performed.

[0174] It should be noted that the computer readable medium shown in the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium that can send, propagate or transmit the program for use by or in connection with an instruction execution system, device or apparatus. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF or the like, or any suitable combination of the above.

[0175] The computer program product of the present application can include a computer readable medium, which can be a built-in medium installed in the device or an external medium such as a USB flash drive, a Blu-ray disc or an SD memory card, or a combination thereof. The computer readable medium has a program stored thereon, and the program can include: an instance segmentation model for performing instance segmentation on a to-be-detected image to obtain each instance in the to-be-detected image; a center calculation module for calculating a preliminary center of each instance based on point cloud data and the each instance; a center correction model for correcting the preliminary center of each instance to obtain a corrected center of each instance; and a target detection model for inputting the corrected center of each instance to output a bounding box and a category of each instance.

[0176] The modules described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware. The described modules can also be arranged in a processor, for example, a processor can be described as: a processor includes a segmentation module, a calculation module, a correction module and a detection module, and in some cases, the names of these modules do not constitute a limitation on the modules themselves.

[0177] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, and when the one or more programs are executed by the device, the device implements the following method: performing instance segmentation on a to-be-detected image using an image instance segmentation model to obtain each instance in the to-be-detected image; calculating a preliminary center of each instance based on point cloud data and the each instance; correcting the preliminary center of each instance using an instance center correction model to obtain a corrected center of each instance; and inputting the corrected center of each instance into a target detection model to output a bounding box and a category of each instance.

[0178] According to the technical scheme of the embodiment of the present application, the technical means that the image instance segmentation model is adopted to perform instance segmentation on the to-be-detected image, the instance center correction model is adopted to correct the preliminary center of each instance, and the corrected center of each instance is input into the target detection model, so the technical problem of poor image detection performance in the prior art is overcome. The embodiment of the present application uses the instance segmentation result of the image to perform instance-level labeling on each point in the point cloud, further corrects the center of each instance, and then performs target detection by using the point cloud detection algorithm. Since the instance information of the point cloud is obtained before being input into the point cloud detection algorithm, the learning difficulty of the detection algorithm is greatly reduced, so the detection performance is greatly improved.

[0179] The foregoing detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and replacements can occur depending on design requirements and other factors. Any modification, equivalent replacement, and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An image detection method characterized by, The method comprises the following steps: obtaining sample images at various angles, marking the positions, categories and contours of various sample instances on the sample images at various angles, and training a first model to obtain an image instance segmentation model; calculating preliminary centers of the various sample instances, marking the bounding boxes of the various sample instances in sample point cloud data, and training a second model to obtain an instance center correction model; inputting attribute data of each point cloud of the various sample instances and the bounding boxes of the various sample instances into a third model for training, so as to train a target detection model; wherein the attribute data of each point cloud comprises three-dimensional coordinates, a category and a center coordinate of a bounding box to which the point cloud belongs; performing instance segmentation on each of the various angle images to be detected by using the image instance segmentation model, so as to obtain various instances in each image to be detected; wherein each frame of point cloud data corresponds to an image to be detected at various angles; calculating preliminary centers of the various instances based on the point cloud data and the various instances; correcting the preliminary centers of the various instances by using the instance center correction model to obtain corrected centers of the various instances; inputting the corrected centers of the various instances into the target detection model to output the bounding boxes and categories of the various instances.

2. The method of claim 1, wherein, The method for calculating the preliminary centers of the various instances based on the point cloud data and the various instances comprises the following steps: projecting the point cloud data to the various instances; merging the point cloud data of the same instance appearing in different images to be detected according to the point cloud data of the overlapping parts in the various instances; calculating the point cloud data of the main body of the various instances by using a clustering algorithm; calculating the preliminary centers of the various instances based on the point cloud data of the main body of the various instances.

3. The method of claim 2, wherein, The method for calculating the point cloud data of the main body of the various instances by using a clustering algorithm comprises the following steps: calculating the point cloud data of the main body of the various instances by using a spatial clustering algorithm, and removing the point cloud data other than the main body of the various instances.

4. The method of claim 1, wherein, The method for training the image instance segmentation model by using the first model comprises the following steps: inputting the sample images at various angles and the positions, categories and contours of various sample instances into the first model for training, so as to train the image instance segmentation model; wherein the first model is Mask R-CNN, Hybrid Task Cascade or BlendMaskd.

5. The method of claim 1, wherein, The method for calculating the preliminary centers of the various sample instances, marking the bounding boxes of the various sample instances in sample point cloud data, and training a second model to obtain an instance center correction model comprises the following steps: calculating the preliminary centers of the various sample instances based on sample point cloud data and the various sample instances; marking the bounding boxes of the various sample instances in the sample point cloud data and calculating the centers of the bounding boxes of the various sample instances; inputting the preliminary centers of the various sample instances and the centers of the bounding boxes of the various sample instances into the second model for training, so as to train the instance center correction model.

6. The method of claim 5, wherein, The method for calculating the preliminary centers of the various sample instances based on sample point cloud data and the various sample instances comprises the following steps: projecting the sample point cloud data to the various sample instances; merge point cloud data of the same sample instance appearing in different sample images according to point cloud data of overlapping parts in the sample instances; calculate point cloud data of the main body of the sample instances by using a clustering algorithm; calculate preliminary centers of the sample instances based on the point cloud data of the main body of the sample instances.

7. An image detection apparatus characterized by comprising: comprise: a segmentation module configured to perform instance segmentation on each angle of the to-be-detected image by using an image instance segmentation model to obtain each instance in each to-be-detected image; wherein each frame of point cloud data corresponds to each angle of the to-be-detected image; a calculation module configured to calculate preliminary centers of the instances based on the point cloud data and the instances; a correction module configured to correct the preliminary centers of the instances by using an instance center correction model to obtain corrected centers of the instances; a detection module configured to input the corrected centers of the instances into a target detection model to output bounding boxes and categories of the instances; a training module configured to: obtain sample images of each angle, mark positions, categories and contours of each sample instance on the sample images of each angle, and train a first model to obtain an image instance segmentation model; calculate preliminary centers of the sample instances, mark bounding boxes of the sample instances in sample point cloud data, and train a second model to obtain an instance center correction model; input attribute data of each point cloud of the sample instances and the bounding boxes of the sample instances into a third model for training, thereby training a target detection model; wherein the attribute data of each point cloud comprises three-dimensional coordinates, a category and a center coordinate of a bounding box to which the point cloud belongs.

8. An electronic device, comprising: comprise: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-6.

9. A computer readable medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the method of any one of claims 1-6.

Images